6 April 2018. Rome. The SCAR Strategic Working Groups ARCH, AKIS and Food Systems organised jointly the Workshop: Programming Research and Innovation for Improved Impact Presentation by Paul Winter

1. Setting the scene
Benjamin Davis
Strategic Programme Leader, Rural Poverty Reduction
Food and Agriculture Organization
Programming Research and
Innovation for Improved Impact
April 6, 2018
Rome

2. Assessing impact
• Wide range of methods
– RCT, quasi/non experimental, simulations, process,
cost-benefit, qualitative, etc.
• Wide range of rigor
– From statistical certainty, to simulating reality (with
more or fewer assumptions), to qualitative depth
• Tons of ongoing research—but how effective?
– Method over potential policy impact
– Studies for publication over real life, less glamorous
challenge of evaluating public sector programmes
(which by definition are messy)
– Ex-post versus ex-ante (and accompaniment)

3. But getting impact from an
impact assessment?
• Challenges of agricultural research
– Focus on narrow adoption, not on welfare impact
– Diffusion of technologies—”Even for innovations with rigorous
evidence demonstrating important gains in the short-run, often
large-scale diffusion does not occur, or, even when it occurred,
has not been maintained” (Macours)
– Little consideration of feasibility for scale up
– Market and policy distortions on supply side
– Move away from focus on yields towards concerns of
marginal/poor farmers—labor saving technologies?
– Is this research what small holder producers want? Co-
generation of knowledge?
• Challenges of measuring impact (at FAO)
• The joint initiative

4. The Transfer Project—a case study on what a
good ex-ante impact assessment entails
• A number of fledgling government programmes and
growing practice in SSA on cash transfers (2008)
• High demand for evidence to answer policy and
programme questions—but little evidence from SSA
• 8 countries: Ethiopia, Ghana, Kenya, Lesotho,
Malawi, South Africa, Zambia and Zimbabwe
• Institutional Partnership between FAO, UNICEF, Save
the Children, University of North Carolina/Chapel
Hill
• Working in close collaboration with national
counterparts, including national governments and
research institutions

5. Transfer Project: objectives
• Provide evidence on the effectiveness of social
cash transfer programs in achieving impacts for
children
• Inform the development and design of social
cash transfer policy and programs
• Promote learning across the continent on the
design and implementation of social cash
transfer evaluations and research

6. Transfer Project: the overall model
• Focus on supporting impact evaluations of national
programmes and research-policy interface
• Impact evaluations as part of broader
evidence/learning agendas and policy processes at
national and regional level
• Mixed methods: quantitative, qualitative and local
economy impacts simulation
• Thematic Innovation:
– Economic and productive impacts (household and local
economy)- led by FAO
– Adolescent well-being and HIV/AIDS- led by UNICEF

7. Overview of impact evaluations
Country
(program)
Targeting
(in addition to poverty)
Sample
size
(HH)
Methodology LEWIE Qual
Years of data
collection
Ghana (LEAP) Elderly, disabled or OVC 1,614 Longitudinal PSM X X 2010, 2012, 2016
Ethiopia (SCTP) Labour-constrained 3,351 Longitudinal PSM X X 2012, 2013, 2014
Kenya (CT-OVC) OVC 1,913 RCT X X 2007, 2009, 2011
Lesotho (CGP) OVC 1,486 RCT X X 2011, 2013
Malawi (SCTP) Labour-constrained 3,500 RCT X X 2011, 2013, 2015
Zambia (CGP) Child 0-5 2,519 RCT
X
X 2010, 2012, 2013, 2014
Zambia (MCTG)
Female, elderly, disabled,
OVC
3,078 RCT 2011, 2013, 2014
Zimbabwe (HSCT)
Food poor, labour-
constrained
3,063
Longitudinal
matched case-
control
X X 2013, 2014, 2016

8. Design of the impact evaluation
• At national level, close collaboration between
policy makers and programme implementers,
development partners and researchers to:
– Identify priority questions and match with
programme objectives
– Dovetail impact evaluation with programme roll-out
• At regional level, community of practice of
researchers, policy makers, and development
partners on objectives, method, implementation
and analysis
• Role of Transfer Project as ‘honest broker’

9. Implementation and Analysis
• Collaborative, consultative process; multiple partners
and stages
• Some key moments:
– Collective definition of timeline
• Training of enumerators; Field work; data entry; analysis-
Baseline and follow-up surveys
– Presentation of findings to key stakeholders
– Baseline: what results can be expected?
– Intermediate products (qualitative, process, simulations, etc)
– Follow-up surveys:
• Did we get the results expected?
• What other unintended impacts can we see?
– Linking of different methods to produce a common story of what
was happening and why

10. Use of Results and Dissemination
• National policy events to present draft and final
results
– With programme implementers, senior officials, and
public events as ongoing process
– Some production of national policy briefs
• Popularization and communication of findings –
video, media, communication materials for
participants
• Regional community of practice
• Other standard stuff

11. Impact of the Transfer Project
• At country level: Results from impact evaluations influenced
design of programs and contributed to strategic policy
decisions
– Adjustment to programme design and implementation (targeting,
transfer size, etc)
– Move from cash to cash+ (specifically in terms of nutrition,
agriculture and HIV/AIDS)- cash is important, but not sufficient
– Expanding the audience for social protection: eg: including
ministries of agriculture and planning
• Contribute to build and strengthen the case for scale-up and
expansion:
– Impact evaluations instrumental in strengthening reputation of
social cash transfer programs, and confidence with which policy-
makers decide scale up
– Economic and productive impacts: addressing concerns regarding
dependency and contribution of the poor to inclusive growth

12. Key factors for success
• Evidence generation imbedded in national policy process,
involving government, national researchers, and international
partners (UNICEF, FAO, etc)
• Timing: Evidence (intermediate products and impact
evaluation) available at critical moments of policy-making
• Solid and technical quality of impact evaluation- credibility of
results
• Learning agenda: impact evaluation and also use of data for
other critical analysis (financing, design, etc)
• Broad scope of the evaluation (consumption, poverty, human
capital, production, labor, community impacts, economic
multiplier) enhanced understanding and appreciation of cash
transfers among a traditionally skeptical audience: social and
economic case
• Government champions, political commitment and influence

13. Insights for innovation in impact assessment
• Align the incentives of researchers, the public
sector and producers
– Reflect upon questions, methods, products, ownership
• Imbed research within broader context
– Economic, social, political cultural
• What else is necessary to make an intervention
successful
• By definition, ex-ante is fundamental
– When feasible