This document outlines the key steps to conduct an impact evaluation of a school feeding program in Mali in 7-8 steps. It involves engaging stakeholders, defining relevant evaluation questions, building a theory of change, defining indicators, designing the evaluation using a randomized controlled trial across treatment and control groups, determining an appropriate sample size, conducting a household survey, and analyzing the collected data. The goal is to evaluate the program's impact on education, nutrition, local agriculture, and welfare outcomes.
3. June mission in Mali
• Government interested in a field experiment
(RCT) using project and control villages
• Government interested in three main issues:
o Good governance: how to make sure that children
are fed?
o Education: what is the impact on attendance
rates?
o Local economy: how the project benefits small
farmers?
5. Impact on education and nutrition
• Impact on enrolment and achievements, and
what is the role played by school quality?
• Does the programme improve attention and
cognition?
• Does the programme improve nutritional status?
Is there catching-up growth?
• What is the extent of substitution effects within
the household?
• What is the impact of the programme on the diet
of the poor?
6. Impact on agriculture
• What is the impact on small farmers in the
short term (incomes) and in the long term
(farm investments)?
• What is the impact of the intervention on
prices and therefore on consumers?
• What is the impact on the wider economy at
the village and regional levels?
14. Intermediate welfare outcomes
• Sometimes welfare outcomes cannot be
observed because:
• Occur in the very long term (example increase in
employment)
• Are not observable (example maternal mortality)
• Intermediate outcomes are used in these
cases: proxy indicators of the final outcomes
along the causal chain
15. Education: outcome indicators
• Enrolment, attendance rates and drop-outs
• Achievement tests (test scores on maths and
language)
• Attention and cognition
17. Food security: outcome indicators
• Full income questionnaire for farmers will
provide data on:
marketed surplus
Farm profits
Technology and capitalisation
Input use
19. The Mali evaluation design
• The government is expanding the intervention
to 60 of the most vulnerable communes
• A commune is an administrative unit
comprising 5 to 15 villages and a similar
number of schools
• The groups considered by the study are:
Control group (no intervention)
Standard school feeding
Home grown school feeding
20. Level 1 comparison: school
feeding-control group
• MOE needs to know the impact of the
intervention on educational indicators
• First comparison is between any school
feeding and a control group with no
intervention
• Outcomes of interests are:
Enrolment rates
Learning achievements
Attention and cognition
Nutritional outcomes
21. Level 2 comparison: school feeding
/home-grown school feeding
• The second comparison is between the
conventional government programme and the
‘home grown’ programme
• Outcomes of interest:
Small farmers’ incomes
Overall programme performance
22. Selection of schools and
communes
• In each of the 60 communes Mayors will
select two school for the intervention of which
one will be randomly assigned to the
programme (pair-matching design or
stratification by commune). A protocol is
designed to avoid contamination.
• Of the 60 communes assigned to the
programme, 30 will be randomly assigned to
the home grown component
24. Sample size
• We collected data from 30 households in each
village:
20 households with children aged 5 to12
10 farmer households (with or without children)
• Sample size is:
1,200 farmer households
3,600 farmer and non-farmer households
6,000 to 7,000 children of primary school age
25. Calculate the sample size
• The size needs to be sufficiently large to
detect the expected effect of the programme
• Detecting sample size is guesswork and the
goal is to produce lower and upper bounds
rather than exact samples
• There is statistical software which is designed
to do this
26. Power
• You need a powerful sample to detect impact
• The power of your sample will be a function of
• The expected programme impact (increasing)
• The variance of the outcome of interest
(decreasing)
• The homogeneity within clusters (decreasing)
• The desired level of Type I error (decreasing)
28. main issues
• Choose the unit of
observation
• Find existing datasets and
surveys
• Establish the timing of data
collection
• Establish whether collecting
cross-section or panel data
• Establish the number of
surveys
• Design the questionnaires
• Administer the survey
29. Choose the unit of observation
• The preferred level of observation is the
‘individual’ or the ‘household’
• Note that individuals are difficult to interview (ex:
consumption data or panel)
• Household is the most frequent unit of observation
• Observations can also be made at cluster level
(village, school, clinic or other)
• Note that sample size will be small
• Data is difficult to collect (who is interviewed?)
30. Data scoping
• Before starting collecting any data you should
first investigate what data and surveys are
available:
• Census data can be used to frame the sample or to
extract control variables
• Existing household surveys (LSMS or DHS) can be used
to form control groups in matching techniques
• Project monitoring data can be used to observe trends
• Survey maybe underway in the same areas. This is
rarely the case, but piggybacking is theoretically
possible
31. Timing of data collection
• 3 main issues to consider:
• How many surveys will be run?
• Baseline, midterm and end-line
• When is the survey starting and for how long?
• Seasonality issues need to be considered
• What is the recall period adopted in the
questionnaire?
• Short recall is more reliable but loses information
32. Cross-section or panel data?
• A cross-section survey collects data from a
population at a point in time
• A panel survey collects data from a population
repeated times
• Panel data are preferable because they
simplify the analysis
• But panel data are not always feasible
• But attrition can be large and there can be
differential attrition
33. Questionnaire design
• Identify the modules that are needed. For
example: a household roster, an education
module, a consumption module etc.
• Look at existing questionnaire designed by
other researchers in similar context
• Examples can be taken from:
• LSMS surveys
• DHS surveys
• Other resources
34. Running a survey in practice
• Hire a firm with the desired capacity
• Ensure enumerators are properly trained and
manuals are available
• Test the questionnaires many times
• Ensure supervision of enumerators in the field
• Ensure households collaborate
• Obtain ethical approval