As part of 2016 IFPRI Egypt Seminar Series on "Improving Nutrition in Egypt"

1. Improving Nutrition in Egypt
What is the Role of Agriculture and
How do we Document Impact?
Jef Leroy & Olivier Ecker
IFPRI, Washington, DC
IFPRI Egypt Seminar Series
Cairo, 16 Nov 2016

2. Overview
• Part 1: Nutrition-sensitive agriculture (Ecker)
• Part 2: Key challenges and solutions in
evaluating the impact of nutrition-sensitive
agricultural programs (Leroy)

3. Overview
• Part 1: Nutrition-sensitive agriculture (Ecker)
• Part 2: Key challenges and solutions in
evaluating the impact of nutrition-sensitive
agricultural programs (Leroy)

4. Agriculture-nutrition linkages: Theory and
evidence
Key message: Agricultural interventions can improve nutrition,
but they do not automatically lead to improved nutrition.
Agricultural programs and projects can improve diets and
nutritional outcomes of farmers and their families through many
different pathways:
1. Increasing farm incomes ( productivity gain)
2. Improving food consumption from own production ( food
quantity and diet diversity)
3. Employment and control over household resources ( e.g.
women empowerment)

5. Agriculture-nutrition linkages: Theory and
evidence
• Growing evidence—from rigorous impact evaluations—
suggest that agricultural interventions can improve (child)
nutrition, if they …
1. Have specific nutrition goals and actions, and
2. Focus on women.
• Many factors influence individual nutritional status (incl. diets,
health, care). Agriculture interventions are part of the
solution, but more is needed.
– E.g., in addition to increasing incomes and improving food
availability and access, behavioral change to promote adequate
nutrition and health practices is needed.

6. Pathways linking agriculture to nutrition
Adapted from:
Gillespie et al.. 2012. The Agriculture-Nutrition Disconnect in India: What Do We Know? IFPRI Discussion Paper 1187. IFPRI, Washington DC.

7. Agriculture-nutrition linkages: Knowledge
gaps and research needs
• Evidence on the impact of agricultural interventions on
nutritional outcomes is still scarce.
• Existing evidence is largely based on small-scale programs in
fairly controlled environments (e.g. home gardens).
• Evidence from large-scale, complex agricultural
interventions—especially for promotion of agricultural
commercialization (“agribusiness”)—is missing so far.

8. Agriculture and nutrition
In summary (Lancet, 2013):
• Agriculture has enormous potential to improve nutrition, but
this potential is yet to be unleashed.
• Lack of critical evidence of impact of agriculture on nutrition is
likely due to:
– Weaknesses in program design, targeting, implementation,
and lack of nutrition goals and interventions, or
– Poor evaluations – design, methods, indicators.

9. Overview
• Part 1: Nutrition-sensitive agriculture (Ecker)
• Part 2: Key challenges and solutions in
evaluating the impact of nutrition-sensitive
agricultural programs (Leroy)

10. Part 2: Key challenges and solutions
1. Challenges in evaluating the impact of
nutrition-sensitive agricultural programs
2. Rigorous evaluation approach to evaluate:
a) Impact
b) Impact pathways, and
c) Cost
3. Recommendations on how to address key
challenges of carrying out sound evaluations
implemented under real life conditions.

11. Part 2: Key challenges and solutions
1. Challenges in evaluating the impact of
nutrition-sensitive agricultural programs
2. Rigorous evaluation approach to evaluate:
a) Impact
b) Impact pathways, and
c) Cost
3. Recommendations on how to address key
challenges of carrying out sound evaluations
implemented under real life conditions.

12. What are some key challenges?
• Complexity of nutrition-sensitive agricultural
programs
• Long impact pathways and time frames
• Trade-off between implementation constraints
and evaluation rigor

13. What are some key challenges?
• Complexity of nutrition-sensitive agricultural
programs
• Long impact pathways and time frames
• Trade-off between implementation constraints
and evaluation rigor

14. Program complexity

15. Complexity nutrition-sensitive
agricultural programs
• Complex in design and implementation:
– Multiple goals, multiple inputs, multiple pathways of
impact, multiple outcomes and impacts;
– Address both the underlying and direct causes of
undernutrition;
– Span across different sectors (e.g. health, agriculture,
education), requiring coordination and integration.
• Within each program intervention potential
variability in:
– Delivery (quantity and quality);
– Utilization;
– Adherence to the program protocol.

16. What are some key challenges?
• Complexity of agricultural programs aimed at
improving nutrition
• Long impact pathways and time frames
• Trade-off between implementation constraints
and evaluation rigor

17. Long impact pathways and time frames
• Long time frames for implementation and evaluation:
– Program development and implementation at desired quality
– Long pathways from program inputs to effects:
E.g.: installing garden beds, preparing the soil, sowing, planting and
harvesting; setting up and implementing the BCC strategy, improving
maternal knowledge through repeated BCC sessions, and achieving
changes in practices;
– Meaningful effect on biological outcomes such as child
anthropometry may require as long as 1,000 days of program
exposure.
– Time to design a rigorous impact evaluation:
Building the program theory framework, developing the evaluation and
sampling design, designing and pre-testing the data collection
instruments, training and standardizing enumerators, planning the
logistics of the field work, enroll the necessary number of study
participants, and seasonality.
• … vs often short time frames imposed by donors

18. What are some key challenges?
• Complexity of agricultural programs aimed at
improving nutrition
• Long impact pathways and time frames
• Trade-off between implementation constraints
and evaluation rigor

19. Implementation constraints vs.
evaluation rigor
Treatment 1
Treatment 2
Treatment 3
Control
Regular program
Cluster randomized controlled design
T24
T18
TNFP
Control
Regular program

20. Part 2: Key challenges and solutions
1. Challenges in evaluating the impact of
nutrition-sensitive agricultural programs
2. Rigorous evaluation approach to evaluate:
a) Impact
b) Impact pathways, and
c) Cost
3. Recommendations on how to address key
challenges of carrying out sound evaluations
implemented under real life conditions.

21. Part 2: Comprehensive evaluation
approach
PROGRAM
Maternal
& child
health &
nutrition
What is the impact
of the program?

22. Impact
evaluation
Part 2: Comprehensive evaluation
approach: embrace complexity
What is the
impact of
the
program?
Process + impact
evaluation
How and why
does the
program (not)
have an
impact?Cost
study
Cost of
the
program?

23. What is the impact of the program?
• impact=(N|with program) - (N|without program)
• Problem: (N|with program) and (N|without
program) never both “observable”
• The key challenge to impact evaluation:
what would have happened in the absence of the
program = counterfactual

24. Creating a valid counterfactual
INTERVENTION CONTROL

25. Experimental designs
• Experimental (or randomized) designs considered
gold standard for impact evaluations.
• Randomization: individual or group (cluster) level.
• *If* done well, one can assume:
– that both groups are comparable;
– that the only difference between the groups is the
program;
– that the control group provides a valid counterfactual
for the intervention group exposed to the program.
differences found in the outcomes of interest
attributable to the program.

26. Experimental designs
• Not always feasible
• Often require creative thinking (oversubscription,
public lottery, delayed intervention, stepped
wedge design, etc.)
• Alternatively: quasi-experimental designs
– use statistical techniques to create a valid comparison
group
– E.g.: propensity score matching (PSM), double
difference (or difference-in-difference) approach,
regression discontinuity, instrumental variable
regressions, etc.
• Stay away from before-and-after or with-and-
without designs!

27. Invalid counterfactuals
68.2
74.8
64.1 64.3
2010 2014
68.2
74.8
64.7
63.0
2010 2014
68.2
74.8
62.3
64.1
2010 2014
68.2
74.8
64.6
65.7
2010 2014
-5.5
-7.1
-5.2
-4.0
-10.0
-5.0
0.0
60.0
65.0
70.0
75.0
80.0
2010-2014 2010-2014 2010-2014 2010-2014
All T24 T18 TNFP
Control Treatment Impact (95% CI)
Impactonstunting(pp)
Wave
68.2
74.8
64.1 64.3
2010 2014
68.2
74.8
64.7
63.0
2010 2014
68.2
74.8
62.3
64.1
2010 2014
68.2
74.8
64.6
65.7
2010 2014
-5.5
-7.1
-5.2
-4.0
-10.0
-5.0
0.0
60.0
65.0
70.0
75.0
80.0
2010-2014 2010-2014 2010-2014 2010-2014
All T24 T18 TNFP
Control Treatment Impact (95% CI)
Impactonstunting(pp)
Wave
Stunting(%)Impact(pp)
0.0
-5.0
Randomized
design
-5.5 pp
68.2
74.8
64.1 64.3
2010 2014
68.2
74.8
64.7
63.0
2010 2014
68.2
74.8
62.3
64.1
2010 2014
68.2
74.8
64.6
65.7
2010 2014
-5.5
-7.1
-5.2
-4.0
-10.0
-5.0
0.0
60.0
65.0
70.0
75.0
80.0
2010-2014 2010-2014 2010-2014 2010-2014
All T24 T18 TNFP
Control Treatment Impact (95% CI)
Impactonstunting(pp) Wave
Before-and-
after
+ 0.2 pp
68.2
74.8
64.1 64.3
2010 2014
68.2
74.8
64.7
63.0
2010 2014
68.2
74.8
62.3
64.1
2010 2014
-5.5
-7.1
-5.2
60.0
65.0
70.0
75.0
80.0
2010-2014 2010-2014 2010-2014
All T24 T18
Control Treatment
Impactonstunting(pp)
Unadjustedstunting(%)
Wave
With-and-
without
- 10.5 pp

28. Impact
evaluation
Part 2: Comprehensive evaluation
approach: embrace complexity
What is the
impact of
the
program?
Process + impact
evaluation
How and why
does the
program (not)
have an
impact?Cost
study
Cost of
the
program?

29. How and why does the program (not)
have an impact?
• Two ways to assess how impact is achieved :
1. Measurement of intermediary measures (outcomes) in impact study
2. Process evaluation study
• Solid understanding of program theory and program impact
pathways is key:
– Key program components,
– Factors that affect optimal delivery or utilization of each component,
– Assumptions associated with each of the components, and
– How the components are expected to be linked (pathways of impact)
• Understanding the pathways to impact:
– Critical to improving program delivery and effectiveness
– Identify what is needed to scale up and to adapt the program in other
settings
– Help interpret (lack of) impact

30. Designing the process evaluation
• Examines the primary inputs, processes, outputs and
outcomes along each of the primary program impact
pathways
• Addresses five key questions:
– Are program services being implemented and provided
according to the program design (inputs and processes)?
– Are program services being utilized as intended (outputs)?
– What is the quality of the program inputs and services (inputs,
processes and outputs)?
– What are the barriers and facilitators to optimal service delivery
and utilization (inputs and processes)?
– Is the program on track to have the desired effect on improving
intermediary outputs and outcomes, such as improvements in
knowledge?

31. Impact
evaluation
Part 2: Comprehensive evaluation
approach: embrace complexity
What is the
impact of
the
program?
Process + impact
evaluation
How and why
does the
program (not)
have an
impact?Cost
study
Cost of
the
program?

32. What is the cost of the program?
• Objectives:
– Estimate the overall cost of the program, the cost of the main program
components, and the program’s cost-effectiveness.
– Estimate savings or cost associated with adding, changing or dropping
program components, adding beneficiaries or scaling up the program.
• Method : Activity Based Costing Ingredients (ABC-I) approach.
• Steps:
– Using the program impact pathways: detailed description of all
program activities.
– Identify the program’s main activities and “ingredients” needed for
each.
– Identify the different types, quantities and costs of the “ingredients”
necessary for each activity.
– Calculate cost of each program activity and of full program.

33. Part 2: Key challenges and solutions
1. Challenges in evaluating the impact of
nutrition-sensitive agricultural programs
2. Rigorous evaluation approach to evaluate:
a) Impact
b) Impact pathways, and
c) Cost
3. Recommendations on how to address key
challenges of carrying out sound evaluations
implemented under real life conditions.

34. Two keys to success
• Solid evaluation framework:
In-depth understanding of the program, program
theory, program impact pathways, rigorous
comprehensive evaluation design
• Strong partnership and collaboration between
implementers and evaluators

35. Strong partnership between implementer
and evaluator
• Establish collaboration at the program design
phase and maintain throughout.
• Align potentially differing priorities, expectations,
incentives and time frames.
• Program implementers:
Share updates and challenges on program roll out and
service delivery
• Evaluators:
Provide regular updates on goals, methods and findings
from their evaluation activities.

36. In summary
• Evidence of what works, how and at what cost is extremely
limited.
• Guidance for future investments requires strong evidence
from rigorous, theory-based comprehensive evaluations of
different nutrition-sensitive program models that bring
together interventions from a variety of sectors (e.g.
health, education, agriculture, social protection, women’s
empowerment, water and sanitation, etc.)
• This presentation focused on:
– How to design and carry out rigorous impact, process, and cost
evaluations.
– How to address some of the perceived insurmountable
challenges that have prevented investments in rigorous
evaluations of such programs in the past.

37. Exciting recent and ongoing studies
• Completed:
– Homestead food production:
• Burkina Faso
• Improved women’s and child’s health, nutrition
• Improved women’s empowerment
– HarvestPlus biofortified orange-fleshed sweet
potato (OSP):
• OSP vines were given to 24,000 households in Uganda
and Mozambique from 2006 to 2009
• project included agriculture component and nutrition
behavior change
• Project increased vitamin A intake of young children

38. Exciting recent and ongoing studies
• Ongoing:
– Improve chicken value chains (Burkina Faso)
– Improve quality of milk sold by informal vendors
(Nairobi, Kenya)
– Reduce aflatoxin in maize (Kenya)
– Improve milk contracts for nomadic farmers
(Senegal)
– USAID-funded, large-scale agribusiness program in
horticulture and a basic health care program
(Upper Egypt)