O slideshow foi denunciado.
Seu SlideShare está sendo baixado. ×

Project from Start to Finish

Carregando em…3

Confira estes a seguir

1 de 23 Anúncio
1 de 23 Anúncio

Mais Conteúdo rRelacionado


Project from Start to Finish

  1. 1. Project Start to Finish: Evaluating education interventions in Haryana Harini Kannan J-PAL South Asia at IFMR Monitoring & Evaluation Training Course for the Indian Economic Service August 13, 2014
  2. 2. Recap of course content 1. What is evaluation? 2. Measuring impacts (outcomes, indicators) 3. Evaluation Methods 4. How to randomize? 5. Sampling and Sample Size 6. Threats and Analysis 7. Project from Start to Finish
  3. 3. Today’s presentation  Background/ Motivation  Intervention: Design, Measurement, Threats  Summary of Research Findings  Next steps
  4. 4. Background  India has achieved substantial success in meeting basic challenge of school enrolment and considerable progress on school facilities and other inputs  Under RTE Act (2009) every child up to 14 years is guaranteed free and compulsory education. No detention & no expulsion policy. No board examination until passing 10th  96% of children (ages 6-14) are enrolled in school. Next challenge is education quality and improving learning levels  Being in school does not imply learning. High variation in students’ preparedness. 53% in grade 5 cannot read a grade 2 level text and 47% cannot do basic arithmetic  The 12th Plan recognizes the centrality of this challenge and has explicitly committed to a target of: "Improving learning outcomes that are measured, monitored, and reported independently at all levels of school education with a special focus on ensuring that all children master basic reading and numeracy skills by class 2 and skills of critical thinking, expression and problem solving by class 5.”
  5. 5. Background  The Problem? • Lack of inputs • Shortage of teachers • Pedagogy/ prescribed curriculum • Teaching/ top of the class • Lack of demand • Health barriers  Previous studies to improve learning • Free books, uniforms • Improving school buildings • Cash grants/ merit scholarships • PTR reductions: more teachers • Remedial tutoring (by Balsakhis) • After school volunteer classes • Summer Learning Camps • Information campaigns • Community participation Teaching at the Right Level
  6. 6. Motivation Behind Current Evaluation  “Teaching at the Right level” is an important policy recommendation from J-PAL • Program shown to work when implemented by Pratham volunteers Opportunity to evaluate a different delivery mechanism • Cost-effectiveness, sustainability considerations and scaling up through government infrastructure  CCE is a nationally mandated policy under RTE • All states have to implement this by 2015 • Request for evaluation from interested government partner Opportunity to sensitize policy makers to “quality evaluations”
  7. 7. Details of CCE & LEP Program Goals Key Components Implementation CCE • Holistic development’ • Regular tracking • Cater to specific needs • Periodic tests, workbooks, project work, class participation • Grades/descriptive remarks • Monthly/quarterly evaluations; bi-annual Report Cards • Teacher training • Provision of manual/materials • Monitoring & mentoring LEP • Need-appropriate education • Focus on lowest performing children • Improve basic Hindi reading levels • Students in grades 3-5 classified in “levels” • Classes restructured according to “levels” • Designing teaching activities “level-wise” • Teacher training • Provision of LEP teaching tools • Assessment to create “levels” • Monitoring & mentoring
  8. 8. Setting  Study Areas • 2 districts – Kurukshetra & Mahendragarh • Chosen in consultation with the GoH Student Learning Outcomes Economic Development Kurukshetra Low High Mahendraga rh High Low
  9. 9. Timeline of Activities QRM: Quarterly Review Meeting (State level) MRM: Monthly Review Meeting (District level) ABRC: Assistant Block Resource Coordinator
  10. 10. Stakeholders Student & Parents Teachers Final Beneficiaries District and Block level functionaries Pratham Field Implementation personnel Sarva Siksha Abhiyan Parishad Department of Elementary Education SCERT Pratham Field Level Implementers Implementers Conceptualizer s Immediate Suppliers
  11. 11. Theory of Change Improve student learning outcomes Low learning outcomes of students High-stakes terminal exams do not allow for feedback Need to emphasize “holistic” development of children Teachers focus on “completing syllabus” – teach to top of the class CCE Formative evaluations Allows for continuous feedback and tracking of performance LEP Pedagogical tool aimed at “teaching according to student ability” Teachers evaluate students frequently and collect data on student performance Teachers use this information to identify low performing students Teacher devices ways in which to help low performing students Teachers conduct quick assessment to classify students according to ability Teachers teach according to the prescribed pedagogy and practices
  12. 12. Research Questions  CCE leads to improvement in student learning outcomes • An important unaddressed constraint has been the lack of student performance feedback .  LEP leads to improvement in student learning outcomes • LEP provides a practical tool to ensure students “catch-up”  Does a combination of the programs lead to larger improvement in student learning outcomes in relation to individual programs • Programs are complementary  What are the various channels through which these programs impact student learning outcomes ?
  13. 13. Intervention Teachers are trained on the CCE & LEP pedagogy and practices Schools are provided the required materials to implement the programs Teachers implement CCE & LEP School-level field monitors monitor the implementation of both programs CCE implemented for classes 1 to 8 LEP implemented for classes 3 to 5 Effective intervention period July 2012 to March 2013
  14. 14. Research Design  What is our sample? • Which schools/classes • How many  How many treatment arms?  Unit of randomization • Student level? • School level? 400 Government Primary Schools 25% CCE & LEP 25 % LEP only 25 % CCE only 25 % Control Power Calculations Level of significance – 95% Power – 80% Variance – Baseline conducted Clustering – yes, ICC = 0.17 Minimum Detectable Effect = 0.1 SD Sample size = 400 schools, 30 students per school
  15. 15. Measurement During the course of the Pre-intervention intervention Post-intervention When ? Process Monitoring Baseline Endline Teacher training, intervention Student learning outcomes Evaluation practices, teacher attitudes Are teachers trained? Quality of teacher training Teacher knowledge and attitude toward programs Are teachers implementing the programs as prescribed? Student learning outcomes Evaluation practices, teacher attitudes, Implementation indicators What? Students tested individually – written and oral Hindi and Math tests Survey of teachers and headmasters Students tested individually Survey of teachers and headmasters Observation of training, teacher surveys pre & post training Interviews with teachers, classroom observations, physical checks of student and school records How?
  16. 16. Threats and Analysis  Low level of implementation • Set-up strong school-level monitoring system  Behavioral changes of Control group (John Henry effect) • Advent of CCE was widely publicized in the local press - Teachers not in CCE schools sought information on the program  Instructions to schools from administration regarding implementation  Contamination • Large scale government teacher training – need to restrict training to the “correct” teachers  Closely monitored this • Teacher transfers between different types of schools  Closely monitored this • The same school monitor visits different types of schools  Extensive training on RCT, contamination and integrity of study design  Attrition • Spend a lot of time/money to track down students at endline • Attrition check done regularly during survey (less than 5% attrition)  Dissemination of results • Nuanced exposition of programs
  17. 17. Hindi Testing Tool
  18. 18. Math Testing Tool
  19. 19. Preliminary Results VARIABLES Hindi Math CCE 0.00229 0.00746 LEP 0.153*** -0.0110 Baseline Oral Hindi score 0.646*** 0.308*** Baseline Oral Math score 0.161*** 0.489*** Female 0.108*** -0.100*** Grade/Class at baseline 0.0655*** 0.0668*** Age at endline (months) -0.00343*** - 0.000721 Constant 0.138*** -0.0547 Observations 11,963 11,950 R-squared 0.635 0.648 Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 Includes dummies for missing baseline scores and age, block*(campus type)*(scoring group) fixed effects (not reported)
  20. 20. Preliminary Results 50.5% Mean scores obtained by students – Hindi and Math Oral tests Control CCE LEP IMPACT 50.6% 50.2% 50.0% 55.5% 49.7% Hindi Math
  21. 21. Data from process monitoring CCE LEP • Exposure to training extensive – around 94% teachers trained . • About 59 % of teachers do have LEP manual with them. • Around 96% of LEP schools have the assessment recording chart with them. • Around 89 % of schools who have the chart have completed documentation correctly • In 92 % of LEP classrooms, teachers teaching according to “levels” • Games observed to be played in over 80% of LEP classes. 70 % of games played appropriate to “levels”. • Exposure to CCE training has been extensive – 92% of teachers are trained. Attendance spotty. • About 48 % of teachers do have the CCE manual. • 35 % of CCE schools do not maintain student Evaluation Sheets. • Of the schools that maintain evaluation sheets, only 57 % could show evidence of such records being maintained. • Around 22 % of headmasters who had problems implementing CCE feel overburdened while 18 % of teachers say that it is too time consuming. • Around 18 % of headmasters who had problems implementing CCE also state that the CCE guidelines are not clear.
  22. 22. Interpreting Results  Need to focus on basic skills: commit to the idea that every child can master them as long as he or she, and the teacher, expend enough effort on it  Large potential gains from reorganizing curricula and classrooms to allow children to learn at their own pace  Programs designed to teach at the right level work, even when implemented by government teachers during the school day
  23. 23. Scale-up  Policy implementation pilot across 2 blocks in Gujarat • Teaching Gujarati and Math • Program monitored by govt. cluster coordinators  Potentially evaluating a “tweaked” program in Tamil Nadu