3. So what was different with Progesa?
• Why was it different?
• Why did policy
makers take note?
• ……
• It was an IMPACT
EVALUATION!
• ….
• What is an IMPACT
EVALUATION??
4. What do we need to measure impact?
Providing sports
programs to
disadvantaged youth
Before After
Project (treatment) 92
comparison
5. Before versus after single difference comparison
Before versus after = 92 – 40 = 52
Before After
Project (treatment) 40 92
comparison
The project led a larger number of girls and
boys to higher levels of health.
But this could have happened in the absence of the program?!
6. Post-treatment comparison
Single difference = 92 – 84 = 8
Before After
Project (treatment) 92
comparison 84
But we don’t know if they were similar before…
7. What do we need to measure impact?
Before After
Project
Comparison
8. Basic Setup of a Randomized Evaluation
Potential Participants
Evaluation Sample
Random Assignment
Treatment
Group Comparison
Group
Participants No-Shows
Based on Orr (1999)
10. Evidence on the impact of
sport on development has
been scarce.
11. An impact evaluation examining the
impact of sport
Did the program cause an
increase in development
indicators amongst program
participants?
Were the effects on boys and
girls the same?
What are the
necessary/sufficient conditions
for success?
Can this be extrapolated to
other circumstances/contexts?
12. 2010 study on sports - Conclusions
Does sport contribute to
personal development and well
being of disadvantaged youth?
•Unclear and non-significant
impact on self-esteem and self-
efficacy.
•Peer leaders (South Africa: yes,
Magic Bus: unclear)
•HIV/AIDS, gender beliefs?
13. Evaluating Sports and development : A need for
better thinking
• Ill defined program theory.
• Small sample sizes
• Survey samples powered
for heterogeneous impacts
(e.g. girls vs. boys, starting
levels of self-esteem)
• Before vs. after but
comparisons not chosen
well.
15. 3ie
Improv
ing the
supply
side an
the dem d
and
side of
eviden
ce base
develo d
pment
s
16. 3ie sub-grant windows
• Funds ‘IMPACT’
evaluations and provides
technical expertise;
• Necessary attributes:
- Counterfactual analysis
- Policy relevant, LIMCs
- Good program theory
- Capacity development
- High technical quality
- Use mixed methods
- Policy impact and
potential for scaling up
21. Bridging the gap: Bringing facts to
policy makers and program managers
• School based deworming:
cuts absenteeism by one-
fourth and costs only 50
cents/child/year.
• Conditional cash transfer
programs: reduce poverty
and increase level of
secondary school
enrollment.
• Financial performance
incentives: can improve
use and quality of health
services.
22. Policy Window
Policy window grants
Preparation grants:
Sri Lanka, Philippines, Fiji,
Ecuador, South Africa,
NEPAD, Uganda etc.
23. 3ie: State of the art expertise
• More than 100 impact
evaluations.
• Systematic reviews
• Policy impact
• More than 50 countries
• Other:
– Staff expertise
– Experts network
– Training
– Monitoring
– In-house research
24. Impact Evaluations
– Efficacy and design
– Effectiveness of
interventions.
– Cost effectiveness of
interventions
– Trade-offs and
unintended
consequences.
3ie was set up in 2008 after the report ‘When will we ever learn’? Was put out by the Center for Global development. The organization’s main mission is to increase development effectiveness through the better use of evidence in developing countries. Our main work is in encouraging impact evaluations and systematic reviews. 3ie provides mainly technical, management and financial support, provided by our donors to high quality impact evaluations and systematic reviews. We disburse approximately XX dollars annually. We have three offices with our headquarters in Delhi and two offices in London and another in Washington DC.
Policy Window grants: India: $500k, breakthrough, early child marriage. China: 2 awards total $1 Million; Intestinal parasites. Uganda: Phase 1: $60k (total to be determined, expected approx US$300k): Staff recruitment and retention in post conflict areas. Morocco: US$330k; youth training. Policy preparation grants: Fiji, ongoing - poverty reduction strategy. South Africa, completed: School feeding programme and ECD Ecuador, 2 proposals submitted: Philippines, on-going with the department of education. Sri Lanka, award in process – to be determined. Others in play Argentina (CIPPEC) Kyrgyzstan (doubtful) FINO (India) NEPAD, e-Africa programme IFRC, Haiti Pakistan, Planning Commission Colombia, SINERGIA Also under discussion a 'Philippines Policy Window' of 10-12 studies financed by AusAID.
Also heterogeneity; What can be scaled up? What can be/should be replicated? Policy impact and relevance.