1. OERC RESEARCH RELATED
TO STUDENT GROWTH
MEASURES AND EDUCATOR
EFFECTIVENESS
Jill Lindsey, Ph.D.
Wright State University
Marsha Lewis, Ph.D.
Ohio University
E x t e n d i n g Y o u r K n o wl e d g e T h r o u g h R e s e a r c h T h a t W o r k s ! I C o l u m b u s , O H I J u n e 1 8 , 2 0 1 4

2.  OERC can examine statewide policy and
practice questions
 Follow and document early implementation
in order to inform policy and practice
 Research implementation for multiple years
as start-up issues are resolved and
implementation takes hold—look for what is
working and what can be improved.
PURPOSE OF RESEARCH

3.  Time span of findings offer insight into changing landscape
around teacher evaluation and student growth measures
 2012: Teachers and principals philosophically supportive of
new evaluation system and need to measure student growth
 2013: HB 555
 2014: Teachers and principals far less supportive and
troubled by concerns related to use of different types of
SGMs for evaluation
 Common Themes
3
FOUR STUDIES SPANNING
THREE YEARS

4. 4
METHODOLOGY
 Structured interviews with superintendents
and administration team members
 Focus groups with teachers
 Surveys of teachers in each pilot LEA
 eTPES data analysis

5.  OTES/OPES Implementation Study (37 LEAs)
 Extended Testing for Value-Added Reporting (23
LEAs)
 Initial Use of Student Learning Objectives (30 LEAs)
 Student Growth Measures Policy & Practice (13
LEAs)
5
FUNDED PROJECTS RELATED TO
STUDENT GROWTH MEASURES

6.  Sequencing, planning, feedback, and student
growth measures for teachers and principals
 Preparation for evaluation
 Experiences of teachers and principals evaluated
using student growth measures
 Processes and measures of student growth that
districts adopted for use in teacher and principal
evaluation systems
6
OTES/OPES IMPLEMENTATION
STUDY

7. 7
EARLIEST FINDINGS
 Generally positive about the new evaluation systems
 Supported use of student growth measures in evaluation
 Lack of trust & misunderstandings about value-added, vendor
assessments, and local measures of student growth
 Unfairness of using different kinds of measures and differing
time cycles for different measures of student growth
 Conversations around new evaluation system focus on
instruction
 Time required to complete evaluation took time away from
working with students
 Appreciative at being asked about their experiences and views

8.  Grant funds provided vendor testing for grades 1,2,3, and
high school subjects
 Provided teacher-level value-added scores from vendor test
results
 Processes and challenges related to extended testing
implementation
 Role of roster verification
 Use of SGMs in educator evaluation
 Best practices/lessons learned
8
SGM EXTENDED TESTING MINI-
GRANT

9. Findings
 Want reliable student
growth measures
 Lack assessment literacy
 Unclear how vendors will
provide data
 Uncertain of roster
verification timing and
impact on VAM
 LEAs opting to use lowest
percentages in weights
 Grateful for being asked
about their experiences
Nine Drop-outs’ Reasons
 Requirement to use
extended testing results
too soon, unfair, not
part of grant
 Cost of extended testing
was too high
 Too many changes, and
too much on teachers’
plates
SGM MINI–GRANTS FOR
EXTENDED TESTING

10. 10
INITIAL USE OF STUDENT
LEARNING OBJECTIVES
Study examined fidelity of SLO use for:
 improving student performance
 measuring academic growth
 evaluating teachers

11.  Training was not uniform across the state
 Assessments varied widely across grade
levels, buildings, and districts
 Processes excessively time-consuming
 More challenging for semester or quarter
courses; limited time to complete the pre-
test, teach, post-test cycle
 Implementation hampered by too many
changes to common core, piloting new
state tests and PARCC, and implementing
OTES
 Many emotional moments and gratitude for
being invited to talk about experiences
Interviews
Surveys
Documents
eTPES
Data
11
EARLY
SLO
THEMES
ALL DATA
NOT YET
ANALYZED

12. 12
SGM POLICY AND PRACTICE
STUDY
OERC study of early adopter districts of
Student Growth Measures
 Designed to provide timely data to inform state
policy and district practice.
 “What does this look like when implemented?”
 Teachers’ perceptions of SGM components
 Do SGMs correlate with Performance on Standards? If
not, why not?
 The distribution of teacher and principal ratings

13. Focus group themes:
 Fairness questions (e.g. Category A teachers do
not know OAA items in advance while Category C
teachers develop their own assessments)
 Principals’ time consumed with teacher observation
activities
 Teachers have questions/misconceptions about
value-added methodology
13
SGM POLICY AND PRACTICE
STUDY

14. 14
SGM POLICY AND PRACTICE
SURVEY
 Deployed late February through mid-April, 2014
 22% response rate (603 teacher respondents/2,709 full-
time teachers) N = 469–classroom teachers, 97
intervention specialists
 Survey responses were similar to focus group findings
 Of the four SGMs (Value-Added, SLOs, Vendor-Approved
Assessments, Shared Attribution), more surveyed
teachers think Student Learning Objectives “most
accurately assess a teacher’s instructional impact.”

15.  Early stage of implementation
 Uncontrollable factors
 Unequal measures/accuracy of the measures
 SLOs teacher-developed, validity/reliability questions
 Others see the SLOs as most fair because focused on
the content taught and results during evaluation year
 Approved vendor assessments may not match content
standards
 Value-added model was not formulated to measure
individual teacher effectiveness
15
FAIRNESS CONCERNS WITH
SGMS & EVALUATION

16. Teachers who see value in SGMs:
 Feel it is important to measure student growth
 Recognize the need for accountability
 SGMs useful source of feedback for planning and
adjustment to outcomes
16
SGM POLICY AND PRACTICE
STUDY

17. “I do think it is important to make sure a child makes
adequate growth. However, there are factors that are
out of my control (attendance, home support, etc.)
that affect a child's learning and are not considered
when calculating the yearly academic growth of a
student.”
“It shows the effectiveness of a teacher and useful
data to adjust your teaching.”
17
SGM POLICY AND PRACTICE
STUDY

18. Teacher–Student Data Link/Roster Verification is necessary to
ensure SGM data quality.
Research Questions:
 Are teachers actively participating in the verification of their
own rosters and percentage of instructional time with
students as specified by Ohio’s roster verification process
guidelines?
 Do principals and teachers have access to adequate
training and technical assistance?
 Do principals and/or teachers perceive any issues with
roster verification?
 What do Ohio educators view as the hallmarks of a good
system?
18
TEACHER ROSTER
VERIFICATION RESEARCH

19. 19
TEACHER ROSTER
VERIFICATION SURVEY
Sent online survey to all teachers and principals who
completed the link/roster verification process in spring
2013 and spring 2014
2013 survey: 5,984 teacher responses from 695 LEAs
2014 survey to-date: 6,778 teacher responses. Survey
still in field.

20. 2011 2013 2014*
(prelim.)
Yes 46% 57% 59%
No 23% 25% 25%
Don't
know
31% 17% 16%
Teachers – Do you think the linkage process accurately
captured what was happening in your classroom (i.e.
students you taught last year, their length of enrollment,
and your percentage of instructional time with them)?
20
TEACHER ROSTER VERIFICATION SUR

21. 21
TEACHER ROSTER
VERIFICATION SURVEY
For teachers that answered “No”:
Teachers – Explain why you think the student–teacher
linkage process did not accurately capture what was
happening in your classroom. (open-ended)
Themes:
 Difficulty dividing time in various co-teaching
situations.
 Unable to account for student absences
 Teachers want to be able to report finer
increments of shared instructional responsibility
 Students’ schedules changed too
often/environment too dynamic to accurately
estimate time

22. 2011 2013 2014*
(prelim.)
Not at all
confident
39% 32% 35%
Somewhat
confident
55% 61% 58%
Very
confident
6% 8% 7%
Teachers – Given your experience with the linkage process, how
confident are you that the linkage process improves the
accuracy of the teacher-level value-added data?
22
TEACHER ROSTER VERIFICATION SURVEY

23. Concerns
 Early in
implementation—lack of
trust and
misunderstandings
 Perceived unfairness of
different kinds of
measures
 Time required to
complete evaluation
 Too many changes at
the same time
Kudos
 Support measuring
student growth
 Training for assessment
literacy desired
 Appreciate being
consulted and heard
 Roster verification
process is improving
23
COMMON THEMES ACROSS TIME

24.  Build trust by continuing to include teachers and
administrators in conversations and policies that impact
them
 Acknowledge concerns as legitimate
 Provide professional development opportunities to
correct misunderstandings and knowledge deficits
 Streamline paperwork where possible; use adaptations
from the field
 Modify policy and roll-out timelines when possible
24
RECOMMENDATIONS

25. QUESTIONS?
JILL.LINDSEY@WRIGHT.EDU
LEWISM5@OHIO.EDU
connect@oerc.osu.edu | oerc.osu.edu

27. PLANNING FOR THE FUTURE,
LEARNING FROM THE PAST: WHAT
CAN SCHOOLS LEARN FROM COLLEGE
AND CAREER PROFILES OF
GRADUATES?
Joshua D. Hawley
Director, OERC and Associate Professor
John Glenn School of Public Affairs
The Ohio State University
Making Research Work for Education

28. Ohio’s Constitution
A “thorough” and
“efficient“
education.
OHIO’S STANDARDS OF PUBLIC
EDUCATION

29. College for all
High
School
College Work
Education and Career
High
School
• CTE
• STEM
College
• AP/Dual
Enrollment
• School +
Work
Workforce
Training
• Apprenticeship
• Military
29
LINKING SCHOOL TO
WORK

30. 30
CHANGING VIEWS OF WORK
REQUIRE NEW INFORMATION
 As educators, we want to know about a variety of
educational outcomes, not just college-going rates
for students. At its broadest, we might consider the
following domains:
 College (traditional two and four-year sectors)
 Credentialed and non credentialed workforce training
 Apprenticeships
 Military

31. 31
QUALITY OF OUTCOMES MATTER
 In this day and age, the quality of the outcomes matter a
great deal, and, therefore, we are concerned with how well
students are prepared to perform over time.
 Concerns we typically have in this case:
 Student remediation
 Does student knowledge match what is required in college
classes?
 Are students prepared to pick a career? (I distinguish this
from a job)
 What happens to kids that go directly from high school to work?
 What happens to kids that dropout (both in terms of further
education and work?)

32. 32
PILOT HIGH SCHOOL REPORTS
 Using data from the Ohio Longitudinal Data Archive
(OLDA), the OERC has been able to answer many of
these questions, beginning for high schools, and
present them in a format that schools can use.
 The report has four key question areas:
 What are the employment outcomes of high school graduates?
 What are the post secondary education outcomes of high school
graduates?
 What is the quality of post secondary education high school
graduates are carrying out?
 What happens to individuals that do not graduate from high
school (dropout)?

33. 33
PILOT REPORTS: COLLEGE AND
CAREER

34. 34
WHAT KIND OF EMPLOYMENT
EXPERIENCES DO STUDENTS HAVE
AFTER SCHOOL?

35. 35
WHAT HIGHER EDUCATION
OUTCOMES DO STUDENTS
HAVE?

36. 36
HOW WELL DO SCHOOLS
COMPARE WITH EACH OTHER IN
REMEDIAL EDUCATION?

37. 37
VISUAL LOOK
College and Career Report
High School Name
County
State
How do X year graduates from this school compare to others in Ohio?
Education Career Outcomes
(Note): Following legal agreement covering data use by the OERC, individual cells with fewer then 10 people have been redacted. They are indicated by a *.
School District State
Number of Students that Started High School in X year Number of Students working in Ohio in the year following graduation
Number of Graduates Average annual earnings for individuals with high school diploma
Number of Dropouts Average annual earnings for individuals without a high school diploma
Average High School GPA
Percent of students in this class eligible for free or reduced lunch Industry of employment for graduates of high school not in college
Retail 20
Average junior year ACT scores for this class by subject Constructi 20
Financial S 60
English
Math
Reading
Science
Composite
Percent of graduates ready for college Industry of employment for high school dropouts
Retail 40
Average state scholarship awards for graduates Constructi 40
Financial S 10
Overall post secondary rate for graduates from this class Other 10
Two year college
Four year college
Other vocational/workforce training
Percent of graduates from this class who attended an in‐state college or university
Percent of graduates from this class who attended an out of state college or university
Trend in college going rates for this school vs. state and district Working and school
*Readiness scores are calculated based on the individual ACT score over or below the "Remeditaion Free Standard"
Footer: Describe data origin. Refer to website that summarizes origin.
Retail
Constr
uction
Retail
Construc
tion
Financial
Services
Other

38. 38
COLLEGE OUTCOMES
How do X year graduates from this school compare to others in Ohio?
Education
(Note): Following legal agreement covering data use by the OERC, individual cells with fewer then 10
people have been redacted. They are indicated by a *.
School District State
Number of Students that Started High School in X year
Number of Graduates
Number of Dropouts
Average High School GPA
Percent of students in this class eligible for free or reduced lunch
Average junior year ACT scores for this class by subject
English
Math
Reading
Science
Composite

39. 39
COLLEGE OUTCOMES (CONT.)
Percent of graduates ready for college
Average state scholarship awards for graduates
Overall post secondary rate for graduates from this class
Two year college
Four year college
Other vocational/workforce training
Percent of graduates from this class who attended an in‐state college or university
Percent of graduates from this class who attended an out of state college or university
Trend in college going rates for this school vs. state and district

40. 40
CAREER OUTCOMES
Number of Students working in Ohio in the year following graduation
Average annual earnings for individuals with high school diploma
Average annual earnings for individuals without a high school diploma
Industry of employment for graduates of high school not in college
Retail 20
Constructi 20
Financial S 60
Industry of employment for high school dropouts
Retail 40
Constructi 40
Financial S 10
Other 10
Retail
Constr
uction
Retail
Construc
tion
Financial
Services
Other

41.  Develop a formal high school college and career report for
select districts (next up, Columbus City Schools; Battelle
For Kids)
 Complete Workforce Success Measures Project with the
Office of Workforce Transformation (see OWT website for
introduction: http://workforce.ohio.gov/ ).
 Work with Ohio Department of Education and Board of
Regents to answer questions about employment outcomes
for K-12 and higher education.
41
FUTURE PLANS

42. THANKS FOR YOUR
ATTENTION!
connect@oerc.osu.edu | oerc.osu.edu
SM22

43. Slide 42
SM22 to be consistent with brochure and briefs
www.oerc.osu.edu | connect@oerc.osu.edu
Sunny Munn, 6/13/2013

45. THIRD GRADE READING
GUARANTEE:
A CASE STUDY
Suzanne Franco, Professor, Wright State University
Jarrod Brumbaugh, Principal, Milton-Union Schools
Making Research Work for Education
Extending Your Knowledge Through Resear ch That Works! I Columbus , OH I June 18,
2 0 14

46.  Ohio Third Grade Reading Guarantee (TGRG)
2012
 In 2012–13, 81% of Ohio’s 3rd graders were
proficient or above
 ODE offered competitive funding grant for
developing TGRG 2013–2014 implementation
 OERC funded a case study of a funded TGRG
three-LEA consortium for 2013–2014
BACKGROUND
45

47.  Co-located in Midwestern Ohio but had
not collaborated on previous initiatives
 Orton-Gillingham Multi-Sensory training
and instructional strategies
 Professional Learning Community (PLC)
 Parent Informational Opportunities
46
CONSORTIUM TGRG PLAN

48. 47
CONSORTIUM DEMOGRAPHICS
LEA Typology Report
Card
Rating
2010–
2011
%
passed
3rd grade
reading
2011–
2012
% passed
3rd grade
reading
2012–
2013
Mobilit
y
%
White/
Non-
Hispani
c
%
Econ.
Dis-
advan.
1 2: Rural w/
avg. pvrty &
small ADM
Exc. 87.3 85.9 7.0 97 40
2 4: Small
Town w high
pvrty & avg
ADM
Exc. 90.3 80.2 11.4 87 53
3 2: Rural w/
avg. pvrty &
small ADM
Exc. w/
Distinc-
tion
84.7 84 6.1 98 19

49.  Feedback and buy-in for the training,
implementation, and PLC.
 Progress and monitoring tools used.
 Reading skills improved for On-Target
students. For Not on Target students?
 Percentage of K–3 students Not on Target in
2012-14?
49
RESEARCH QUESTIONS

50. For each LEA:
 Document analysis of historical data and end-of-year
2014 RIMPS
 Interviews and focus groups with Administrators (6),
Teachers (12)
 Observations of O/G training and classroom instruction
50
METHODOLOGY

51. 51
O/G TRAINING
Training Details
 Two 5-day sessions the week after school year ended
 One 5-day session in November
 Refresher course available summer, 2014
Training Feedback
 Teachers felt it was engaging but too long, or covered
grade levels not in their interest. They would like to repeat
after one year of implementation.
 Administrators from one LEA attended training. They felt
that common language helped with classroom observations.

52. 52
IMPLEMENTATION
Implementation Details
 LEA 1 – O/G not required due to receipt of grant
funds in mid-September, 2013. Used in RTI, Title 1,
and other interventions.
 KRAL is the identifier for K; State assessment tool for
grades 1–3
 DIBELS is the progress monitor along with STAR and
Study Island
 LEA 2 – O/G not required (see above). Used in RTI,
intervention, and Title 1
 NWEA (2012) and DIBELS (2013) for K–3
 DIBELS is the progress monitor
 LEA 3 decided not to participate

53. Implementation Feedback Details
 Not all supplies at beginning of year for all teachers due to
delay in receiving grant funds
 Not all training completed at beginning of year (new
teachers)
 Use of O/G not required; inconsistency a challenge for
teams
 Merging O/G with LEA-approved reading curriculum difficult
 Parent Nights were not well attended; PLC not formed
53
IMPLEMENTATION FEEDBACK

54. 54
PROGRESS AND MONITORING
TOOLS
 LEA 1: DIBELS
 LEA 2: NWEA (2012); DIBELS (2013)
Feedback
 O/G assessment tools not Ohio-approved, therefore the
LEAs use DIBELS and NWEA to assess student progress
 RIMPs not standardized among LEAs (issue for moving and
determining LEA or statewide impact)
 For highly mobile student populations, 30-day requirement
for RIMP is very difficult to meet.
 Too much testing for young students; test anxiety rising

55. Successes
 After School
Program
 Students respond
well to Multi-Sensory
 Teachers want more
training
Challenges
 Use of O/G not
consistent
 Costs to sustain
 O/G assessments not
state approved
 RIMP forms could be
improved; data should be
collected for analyses
 No information about
other LEA TGRG plans
55
CONSORTIUM SUMMARY

56.  LEA1
 Grade 3 results to date
 Changes in implementations for 2014–2015
 LEA 2 Details
 Grade 3 results to date
 Changes in implementations for 2014–2015
56
2013–2014 RESULTS
2014–2015 PLANS

57.  Assessment tools aligned with TGRG programs
funded by state need approval.
 Primary students exhibit high anxiety regarding
TGRG, impacting performance and fear of
school.
 Required testing takes away from instruction
time. Embrace testing that collects needed
data for all accountability purposes, not just
one initiative.
57
TESTING RECOMMENDATIONS

58.  Continue funding for TGRG development.
 Continue monitoring LEA implementation of
funded and non-funded TGRG
implementation plans, and share “lessons
learned.”
 Revise RIMP format and collect RIMP data
for longitudinal analyses of common
deficiencies across the state.
58
TGRG POLICY
RECOMMENDATIONS

59. QUESTIONS
suzanne.franco@wright.edu
BrumbaughJ@milton-union.k12.oh.us
thompsond@piqua.org
Making Research Work for Education
Extending Your Knowledge Through Resear ch That Works! I Columbus , OH I June 18,
2 0 14

61. READY OR NOT?
E x t e n d i n g Y o u r K n o wl e d g e T h r o u g h R e s e a r c h T h a t W o r k s ! I C o l u m b u s , O H I J u n e 1 8 , 2 0 1 4
Ginny Rammel, Ph.D.
Superintendent
Milton-Union Exempted Village Schools

62.  Research and study
 Use of multiple forms of data
 Create a culture of “calculated risk-takers”
 Embed professional development
“EVERY STUDENT, EVERY DAY”

63.  Role model to all at all times
 Establish high expectations
 Collaborate, share
 Trust, be truthful and supportive
 Know your staff
63
CULTURE TAKES TIME TO
CHANGE!

64. 64
EXPLORE OPPORTUNITIES
 Through grants, pilot studies, action
research
 Connect with:
 Personnel at colleges and universities, OERC
 Educators from other districts
 Members of professional organizations
 Policymakers, legislators

65.  Milton-Union was involved in a number of
grants:
 RttT Mini-grant Value-Added
 Student Growth Measures
 Early Literacy and Reading Readiness
 OERC case study
The more you and your staff research, study,
and share data, the better decisions you make.
Collaborating and working together help to
create a culture of “Every Student, Every Day.”
65
EMBEDDED PD:
BE AN ACTIVE PARTICIPANT

66. Our culture and the research and data from
our grant involvement led to development
of our OTES instrument, and a successful
year of implementation
This trust and openness flowed throughout
recent negotiations.
66
RESULTS

67. All initiatives impact one another and YOU:
 OTES
 OPES
 Graduation requirements
 Third Grade Reading Guarantee
Keep the main thing the main thing – is
what I’m doing going to help students
learn? Are we preparing students for
“down-the-road” careers?
67
CHANGE WILL OCCUR WITH OR
WITHOUT YOU!

68.  Do the research upfront
 Study the data
 Reflect, revise if necessary
 Building project
 Food service program
68
CALCULATED RISK-TAKERS

69.  All day, every day kindergarten
 On-site Head-Start programs
 Grouping students by quintiles
 H.S. ACT EOC exams
 Recognized as a U.S. Department of Education
Green Ribbon School
 Food service program ended the year in the black!
69
DATA SUPPORTS INITIATIVES

70.  How do we better prepare students for their
futures – colleges, universities, employers?
 How can we convey to young parents the
importance of their role as a teacher?
 How can we differentiate education so all
students are better served?
 How can we better communicate the results
of research and the sharing of data?
70
NEXT STEPS…

71. QUESTIONS?
RAMMELV@MILTON-UNION.K12.OH.US
connect@oerc.osu.edu | oerc.osu.edu