1. OPAPP and FAMS: Two Studies of
Interim Assessments
Shazia R. Miller
Copyright © 2011 Managing Director
American Institutes
for Research
All rights reserved. June 2012

2. Interim Assessments
Introduction
ASSESS
OPAPP
FAMS
Wrap Up
ADJUST
INTERPRET
INSTRUCTION
2

3. Intro
OPAPP
Make no little plans: they
FAMS have no magic to stir
Wrap Up men’s souls.
Daniel Burnham
3

4. Study 1: Ohio Performance
Assessment Pilot Program (OPAPP)
Intro
Learning Assessment Task Dyad-System
OPAPP  Rich, lengthy complex learning tasks
 Shorter but still complex online
FAMS
assessments
Wrap Up  Scoring of online assessments by
trained teachers
 Re-engagement as appropriate
4

5. Video of a Learning Task
5

6. Summary of Research Questions
1.How is OPAPP implemented?
Intro
2.Does the OPAPP assessment allow students to
demonstrate their skills?
OPAPP 3.What valid inferences do school staff believe can be made
from students’ performance on online assessment tasks?
FAMS 4.Do schools have the required technology?
5.How do students’, teachers’, administrators’ and coaches’
Wrap Up perceptions change?
6.What policy or cultural changes are implemented in schools
(i.e., extra prep time, PD, support, communities of
practice)? Can these be sustained?
7.To what extent do the programs result in improved student
performance?
6

7. OPAPP Data Collection
Intro • Surveys
• Extant Data
OPAPP
• Program Participation
FAMS • Online server logs
• Case Study Interviews
Wrap Up
• Documents
• Video
7

8. OPAPP Classroom Video Footage
Intro
OPAPP
FAMS
Wrap Up
A stationary A hand-held , Interview with
camera. mobile camera the teacher
8

9. Details of Videography
Intro Videography conducted in spring 2012:
• 4 science and 1 math classroom
OPAPP • At 1 high school
• A range of beginning and veteran teachers
FAMS
• One inclusive, special education classroom with
Wrap Up co-teaching
• 1 class period of a learning task (Learning tasks
took pilot teachers anywhere from 3 to 15 class periods to
complete.)
9

10. Paired Group Discussion
Find a partner near you and discuss how
a research could make use of this rich,
but unsystematic data.
10

11. How We Used the Videos:
Modified Cognitive Interviews
Intro • 2 researchers reviewed all clips
OPAPP
• Created sequences focusing on
teacher instructional practice.
FAMS • Reviewed videos with the teachers
Wrap Up  What are you doing now (and why)?
 What are your students doing now
(and why)?
 How typical is this for your class?
 Questions specific to video.
11

12. Study 2: Formative Assessments for
Middle School (FAMS)
Intro • Goal for teachers to develop
OPAPP individual portfolio of methods
FAMS
 Assistance
- Professional development
Wrap Up
- Intensive coaching
- Communities of practice
12

13. Summary of Research Questions
1.What changes are there in perceptions of the program?
Intro
2.What changes are there in policy or culture?
3.To what extent do the programs result in improved student
OPAPP performance?
4.What factors are involved in developing and maintaining
FAMS communities of practice?
5.What are the facilitators and barriers to developing and
Wrap Up maintaining communities of practice?
6.What are the key elements of the coaching program that
make it effective, or prevent it from being effective?
7.How can the coaching component be scaled up?
13

14. Data Collection
Intro •Surveys
OPAPP •Coach logs
•Extant documentation
FAMS
•Interviews
Wrap Up
•Student performance on state
assessments
14

15. FAMS Preliminary Interview Findings
from Two Case Study Schools
Intro Successes
• Teacher growth
OPAPP
• Student intellectual risk taking
FAMS • Self Assessment
Wrap Up
15

16. FAMS Preliminary Interview Findings
from Two Case Study Schools
Intro Challenges
OPAPP
• Program designed to be individualized,
schools take ownership
FAMS • Teachers and coaches sometimes felt
Indiana
there was not enough guidance
Wrap Up
16

17. Intro
OPAPP
Discussion: What are the risks
FAMS and benefits of a individualized
Wrap Up
program?
17

18. Interim Assessments
Intro
OPAPP
FAMS
Wrap Up
18

19. Shazia R. Miller
P: 312-288-7616
E-Mail: smiller@air@air.org
20 N. Wacker Dr, Suite 1231
Chicago, IL 60606
General Information: 800-356-2735
Website: www.air.org
19

20. OPAPP: Who is in the Pilots?
Fall Spring Fall Spring Fall Spring
2011 2012 2012 2013 2013 2014
23 Continuing
Cohort High Schools
1 Grades 11-12 ELA,
math, science
10 New High Schools
Cohort
Grades 9-12
2
ELA, math, science, social studies, career tech
10 New Elementary Schools
Cohort
Grades: 3-5
3
ELA, math, science, social studies
20

21. FAMS: Who is in the Pilots?
Fall Spring Fall Spring Fall Spring
2011 2012 2012 2013 2013 2014
Cohort 6 Schools
1 Grades 5-8
ELA and Math
Cohort 7 Schools
2 Grades 5-8
ELA and Math
Cohort 9 Schools
3 Grades 5-8
ELA and Math
21