ISPC14

1. Standing Panel on Impact Assessment
Doug Gollin, SPIA Chair, 15 Sept 2016

2. Content
http://impact.cgiar.org/
• Work since last ISPC meeting (Lima, May 2016)
• First SIAC synthesis report: The “rigor revolution” in impact
assessment for agricultural research
Causal identification
Measurement
Representativeness of sampling
• Towards a second phase for the SIAC program

3. Recent calendar http://impact.cgiar.org/
June
• 3ie / IFAD workshop “Designing and implementing high-quality, policy-
relevant impact evaluations” (SPIA contribution: DNA fingerprinting into IEs)
• FERDI / SPIA workshop “Agricultural innovation: Learning for adopting”
(Portfolio of 4 SIAC RCTs, linking with other RCTs outside, and thinking
broadly about social sciences across CGIAR + Science of innovation / impact)
• MEL CoP “Taskforce on selection of harmonized indicators” (advice /
caution)
July
• Impact Assessment Focal Point Meeting (our main CoP – identify CAPI need)
• SIAC long-term / large-scale impact assessment studies: Mid-term meeting
(Portfolio of 7 studies, updates, course-correction)
August
• SIAC workshop on innovative methods for measuring adoption of
agricultural technologies (Fingerprinting, remote sensing, SMS and CAPI
surveys - establishing proof of concept and thinking about scaling up)

4. Previous approaches to IA in the CGIAR
http://impact.cgiar.org/
• SPIA core business for a long time was to generate aggregate
estimates of rates of return to investments in the CGIAR
• Methodologically simple but crude
• Appropriate for the era:
• Technologies easily identified
• Impacts largely unidimensional
• Impact modelled as an increase in production multiplied by a
price.
• Consumers and/or producers benefit depending on assumptions
• Extent of broad pool of benefits is a function of:
% adoption
Average productivity gain per unit adoption

5. The ‘rigor revolution’ in IA
http://impact.cgiar.org/
• SPIA commissioned paper to Alain de Janvry and
Elisabeth Sadoulet in 2010
• Their paper “Recent Advances in Impact Analysis
Methods for Ex-post Impact Assessments of
Agricultural Technology: Options for the CGIAR”,
published in 2011 laid foundations for big changes
• Emphasized the need to take seriously that
comparisons of adopters and non-adopters do not
account for all the differences between them,
especially in relation to “unobservables.”
• Recommended a portfolio of RCTs

6. 1. Causal Identification
http://impact.cgiar.org/
• SPIA took on board these recommendations.
• Opted not to focus entirely on RCTs.
• Methodologically agnostic – we want careful and appropriate
combinations of methods
• More complicated question is being clear about which non-
experimental research designs are appropriate, and in what
contexts

7. Institutionalizing new methods
http://impact.cgiar.org/
• Quality-rating system for impact studies / claims in CGIAR
launched earlier this year – no voluntary take-up
Future:
• Shift into regular audits / reviews of impact claims (retrospective)
as well as continuing a forward looking advice function reviewing
research designs for future impact studies (prospective)
• Periodic, predictable synthesis reports on state of knowledge of
impacts

8. 2. Measurement matters
http://impact.cgiar.org/
Adoption
• Often the critical missing data for understanding impacts from
different streams of research
• Definition and measurement of adoption far from simple
Outcome variables
• Productivity (Plot area measurement? Crop-cuts?)
• Remote sensing for environmental benefit streams?
• Data on diets? Anthropometry?

9. Genotype Farmer-elicited name
Maize in Uganda: SPIA / LSMS-ISA /
UBoS / Diversity Arrays
• Data from 540 HHs in 45
enumeration areas
• Enumerators from UBoS trained
for 1 month
• CAPI-based survey + grain-based
highly-quantitative DArTSeq
genotyping
• 2% of farmers were correct about
the variety they were growing

10. Line
transects:
Reference
value
Enumer
ator
with
visual-
aid
Self-
report
with
visual-
aid
Enumer
ator
estimate
Self-
report
Remote-
sensing
Drone
On the ground or from the air? Measuring crop
residue retention, Ethiopia (teff, maize, wheat, barley)

11. 3. Statistical representativeness
http://impact.cgiar.org
• World Bank Living Standards Measurement Study –
Integrated Surveys of Agriculture (LSMS-ISA)
• 8 countries in SSA – all important to CGIAR
• Average of 5,000 HHs / country, nationally
representative
• Panel – visited every 2 years
SPIA role:
• Surveys lack modules / questions on agricultural
technologies (varieties, NRM practices)
• SPIA’s comparative advantage to work to improve
this for benefit of CGIAR as a whole
Future:
• Help bring about a geographic focusing of CGIAR

12. Towards SIAC phase 2: (2018-2022)
http://impact.cgiar.org/
1. Country baselines and monitoring in key geographies
2. Database of claims of policy influence resulting from CGIAR research
3. Maintain focused competitively-commissioned portfolio of ex-post impact
assessments
4. Synthesis reports on predictable and regular production cycle
5. Audits of impact claims
6. Improving the prediction of technology success in farmers’ fields
7. Capacity-building of economics / social science function
8. More work on methods as a CGIAR-wide public good

13. Country baselines and monitoring
http://impact.cgiar.org/
What?
• Proposing 6 high-priority countries + possible spillover countries
• Phased approach between now and 2018/19 for first waves
• Partnerships with national government agencies and local implementing
bodies with expertise in survey management
• Collaboration with CGIAR partners on identifying priorities and piloting
methods
Why SPIA?
• CGIAR-wide public goods
• Established network of potential partners (World Bank LSMS-ISA; IFAD;
Excellence in Breeding - ICRISAT/CIMMYT; Big Data – CIAT/IFPRI; Diversity
Arrays)
• Independence
• Long history of documenting technology adoption

14. SIAC PHASE 1 (piloting now) SIAC PHASE 2? 6 country
CRP total
(M HHs)
Overall
CRP
totalEthiopia Tanzania Uganda India B’desh Vietnam
%
share
A4NH 0.50 0.56 1.80 2.50 3.10 8.5 20.5 41
CCAFS 0.80 0.40 3.00 0.50 1.00 5.7 10.9 53
DCL 8.00 10.00 18.0 25 72
Fish 0.11 1.80 1.9 4.9 39
FTA 4.20 1.20 1.50 6.20 2.60 15.7 41.3 38
Livestock 2.05 1.44 0.31 0.48 0.16 0.11 4.6 6.5 70
Maize 2.50 1.10 0.30 3.70 0.80 8.4 15.0 56
PIM 1.00 5.00 2.00 1.50 9.5 14.5 66
Rice 0.09 4.27 1.19 0.76 6.3 16.5 38
RTB 0.30 0.80 0.30 1.4 8.0 18
Wheat 2.00 0.01 8.00 0.34 10.4 17.2 60
WLE 1.00 0.55 12.50 2.75 16.8 21.0 80
Country total
(M HHs) 22.05 10.76 4.71 52.95 12.14 4.47 107.1 201.2 53
% share of all
rural HHs 127 141 72 29 52 29
N.B. - CRP data from March full proposal versions (not yet checked against revisions)