Project from Start to Finish

1. Project Start to Finish:
Evaluating education interventions in
Haryana
Harini Kannan
J-PAL South Asia at IFMR
Monitoring & Evaluation Training Course for the Indian Economic Service
August 13, 2014

2. Recap of course content
1. What is evaluation?
2. Measuring impacts (outcomes, indicators)
3. Evaluation Methods
4. How to randomize?
5. Sampling and Sample Size
6. Threats and Analysis
7. Project from Start to Finish

3. Today’s presentation
 Background/ Motivation
 Intervention: Design, Measurement, Threats
 Summary of Research Findings
 Next steps

4. Background
 India has achieved substantial success in meeting basic challenge of school
enrolment and considerable progress on school facilities and other inputs
 Under RTE Act (2009) every child up to 14 years is guaranteed free and
compulsory education. No detention & no expulsion policy. No board
examination until passing 10th
 96% of children (ages 6-14) are enrolled in school. Next challenge is education
quality and improving learning levels
 Being in school does not imply learning. High variation in students’
preparedness. 53% in grade 5 cannot read a grade 2 level text and 47% cannot
do basic arithmetic
 The 12th Plan recognizes the centrality of this challenge and has explicitly
committed to a target of: "Improving learning outcomes that are measured,
monitored, and reported independently at all levels of school education with
a special focus on ensuring that all children master basic reading and
numeracy skills by class 2 and skills of critical thinking, expression and
problem solving by class 5.”

5. Background
 The Problem?
• Lack of inputs
• Shortage of teachers
• Pedagogy/ prescribed curriculum
• Teaching/ top of the class
• Lack of demand
• Health barriers
 Previous studies to improve learning
• Free books, uniforms
• Improving school buildings
• Cash grants/ merit scholarships
• PTR reductions: more teachers
• Remedial tutoring (by Balsakhis)
• After school volunteer classes
• Summer Learning Camps
• Information campaigns
• Community participation
Teaching at
the Right
Level

6. Motivation Behind Current Evaluation
 “Teaching at the Right level” is an important policy recommendation
from J-PAL
• Program shown to work when implemented by Pratham volunteers
Opportunity to evaluate a different delivery mechanism
• Cost-effectiveness, sustainability considerations and scaling
up through government infrastructure
 CCE is a nationally mandated policy under RTE
• All states have to implement this by 2015
• Request for evaluation from interested government partner
Opportunity to sensitize policy makers to “quality evaluations”

7. Details of CCE & LEP
Program Goals Key Components Implementation
CCE
• Holistic
development’
• Regular tracking
• Cater to specific
needs
• Periodic tests, workbooks, project
work, class participation
• Grades/descriptive remarks
• Monthly/quarterly evaluations; bi-annual
Report Cards
• Teacher training
• Provision of
manual/materials
• Monitoring & mentoring
LEP
• Need-appropriate
education
• Focus on lowest
performing children
• Improve basic Hindi
reading levels
• Students in grades 3-5 classified in
“levels”
• Classes restructured according to
“levels”
• Designing teaching activities “level-wise”
• Teacher training
• Provision of LEP
teaching tools
• Assessment to create
“levels”
• Monitoring & mentoring

8. Setting
 Study Areas
• 2 districts – Kurukshetra
& Mahendragarh
• Chosen in consultation
with the GoH
Student
Learning
Outcomes
Economic
Development
Kurukshetra Low High
Mahendraga
rh
High Low

9. Timeline of Activities
QRM: Quarterly Review Meeting (State level)
MRM: Monthly Review Meeting (District level)
ABRC: Assistant Block Resource Coordinator

10. Stakeholders
Student
& Parents
Teachers
Final
Beneficiaries
District and Block level
functionaries
Pratham Field
Implementation personnel
Sarva Siksha Abhiyan Parishad
Department of Elementary Education
SCERT
Pratham
Field Level
Implementers
Implementers
Conceptualizer
s
Immediate
Suppliers

11. Theory of Change
Improve
student
learning
outcomes
Low learning
outcomes of
students
High-stakes terminal
exams do not allow
for feedback
Need to emphasize
“holistic”
development of
children
Teachers focus on
“completing syllabus”
– teach to top of the
class
CCE
Formative
evaluations
Allows for
continuous
feedback and
tracking of
performance
LEP
Pedagogical tool
aimed at “teaching
according to student
ability”
Teachers evaluate
students frequently
and collect data on
student performance
Teachers use this
information to
identify low
performing students
Teacher devices
ways in which to
help low performing
students
Teachers conduct
quick assessment to
classify students
according to ability
Teachers teach
according to the
prescribed pedagogy
and practices

12. Research Questions
 CCE leads to improvement in student learning outcomes
• An important unaddressed constraint has been the lack of
student performance feedback .
 LEP leads to improvement in student learning outcomes
• LEP provides a practical tool to ensure students “catch-up”
 Does a combination of the programs lead to larger improvement in
student learning outcomes in relation to individual programs
• Programs are complementary
 What are the various channels through which these programs
impact student learning outcomes ?

13. Intervention
Teachers are trained on the
CCE & LEP pedagogy and
practices
Schools are provided the
required materials to
implement the programs
Teachers implement CCE &
LEP
School-level field monitors
monitor the implementation
of both programs
CCE implemented for classes 1 to 8
LEP implemented for classes 3 to 5
Effective intervention period July 2012 to March 2013

14. Research Design
 What is our sample?
• Which schools/classes
• How many
 How many treatment arms?
 Unit of randomization
• Student level?
• School level?
400
Government
Primary
Schools
25%
CCE & LEP
25 %
LEP only
25 %
CCE only
25 %
Control
Power Calculations
Level of significance – 95%
Power – 80%
Variance – Baseline conducted
Clustering – yes, ICC = 0.17
Minimum Detectable Effect = 0.1 SD
Sample size = 400 schools, 30 students
per school

15. Measurement
During the course of the Pre-intervention intervention Post-intervention
When
?
Process Monitoring
Baseline Endline
Teacher training, intervention
Student learning
outcomes
Evaluation
practices, teacher
attitudes
Are teachers trained?
Quality of teacher training
Teacher knowledge and attitude toward
programs
Are teachers implementing the programs as
prescribed?
Student learning
outcomes
Evaluation
practices, teacher
attitudes,
Implementation
indicators
What?
Students tested
individually –
written and oral
Hindi and Math
tests
Survey of teachers
and headmasters
Students tested
individually
Survey of teachers
and headmasters
Observation of training, teacher surveys pre
& post training
Interviews with teachers, classroom
observations, physical checks of student
and school records
How?

16. Threats and Analysis
 Low level of implementation
• Set-up strong school-level monitoring system
 Behavioral changes of Control group (John Henry effect)
• Advent of CCE was widely publicized in the local press - Teachers not in CCE schools
sought information on the program
 Instructions to schools from administration regarding implementation
 Contamination
• Large scale government teacher training – need to restrict training to the “correct”
teachers
 Closely monitored this
• Teacher transfers between different types of schools
 Closely monitored this
• The same school monitor visits different types of schools
 Extensive training on RCT, contamination and integrity of study design
 Attrition
• Spend a lot of time/money to track down students at endline
• Attrition check done regularly during survey (less than 5% attrition)
 Dissemination of results
• Nuanced exposition of programs

17. Hindi Testing Tool

18. Math Testing Tool

19. Preliminary Results
VARIABLES Hindi Math
CCE 0.00229 0.00746
LEP 0.153*** -0.0110
Baseline Oral Hindi score 0.646*** 0.308***
Baseline Oral Math score 0.161*** 0.489***
Female 0.108*** -0.100***
Grade/Class at baseline 0.0655*** 0.0668***
Age at endline (months) -0.00343***
-
0.000721
Constant 0.138*** -0.0547
Observations 11,963 11,950
R-squared 0.635 0.648
Robust standard errors in parentheses
*** p<0.01, ** p<0.05, * p<0.1
Includes dummies for missing baseline scores and age, block*(campus type)*(scoring
group) fixed effects (not reported)

20. Preliminary Results
50.5%
Mean scores obtained by students –
Hindi and Math Oral tests
Control CCE LEP
IMPACT
50.6% 50.2%
50.0%
55.5%
49.7%
Hindi Math

21. Data from process monitoring
CCE LEP
• Exposure to training extensive – around
94% teachers trained .
• About 59 % of teachers do have LEP
manual with them.
• Around 96% of LEP schools have the
assessment recording chart with them.
• Around 89 % of schools who have the
chart have completed documentation
correctly
• In 92 % of LEP classrooms, teachers
teaching according to “levels”
• Games observed to be played in over
80% of LEP classes. 70 % of games
played appropriate to “levels”.
• Exposure to CCE training has been
extensive – 92% of teachers are trained.
Attendance spotty.
• About 48 % of teachers do have the CCE
manual.
• 35 % of CCE schools do not maintain
student Evaluation Sheets.
• Of the schools that maintain evaluation
sheets, only 57 % could show evidence of
such records being maintained.
• Around 22 % of headmasters who had
problems implementing CCE feel
overburdened while 18 % of teachers say
that it is too time consuming.
• Around 18 % of headmasters who had
problems implementing CCE also state that
the CCE guidelines are not clear.

22. Interpreting Results
 Need to focus on basic skills: commit to the idea that every
child can master them as long as he or she, and the teacher,
expend enough effort on it
 Large potential gains from reorganizing curricula and
classrooms to allow children to learn at their own pace
 Programs designed to teach at the right level work, even
when implemented by government teachers during the school
day

23. Scale-up
 Policy implementation pilot across 2 blocks in Gujarat
• Teaching Gujarati and Math
• Program monitored by govt. cluster coordinators
 Potentially evaluating a “tweaked” program in Tamil Nadu