Presentation by Charlie Crissman

1. Considerations for
AAS CRP Impact Evaluation
Workshop on Strengthening Impact
Evaluation in Natural Resource
Management
WorldFish, Penang, Malaysia
4-5 September 2012
Making a difference in the lives of the poor

2. Pre-implementation evaluation
•Are the objectives well defined so that outcomes can be stated in
measurable terms?
•Is there a coherent and credible implementation plan that provides clear
Performance Logic Chain Evaluation evidence of how implementation is to proceed and how successful
implementation can be distinguished from poor implementation?
The performance logic chain assessment evaluation
•Is the rationale for the deployment of resources clear and commensurate
strategy is used to determine the strength and logic
with the requirements for achieving the stated outcomes?
of the causal model behind the policy, program, or
project.
Process implementation evaluation
What did or did not get implemented as
planned?
Impact Evaluation Rapid Appraisals
… the classic evaluation that provide timely, relevant information to
attempts to find out the decision-makers on pressing issues they face
changes that occurred, and to in the project and program setting.
what they can be attributed
The aim of applied research is . . .
to facilitate a more rational decision-making
process in real-life circumstances
Case Studies
… use when a manager needs in-depth
information to understand more clearly what
happened with a policy, program, or project
Making a difference in the lives of the poor

3. AAS CRP Implementation strategy
Making a difference in the lives of the poor

4. 1. A framework to guide selection of
impact designs
Making a difference in the lives of the poor

5. 2. Evaluation Challenge for AAS
•Many CG stakeholders view experimental designs for IA
as ‘best’
•Mostly experimental or statistical methods based IA NRM-
type programs demonstrate little evidence of impact
•Tradition of CGIAR IA could result in mis-match of
questions and methods that could bias results against AAS
program
•Demonstrate the validity (legitimacy) of alternate methods
for AAS CRP
Making a difference in the lives of the poor

6. 3. Evaluation Goals of AAS
•Accountability
•Respect Consortium and Fund needs for
demonstration of performance and potential for
impact
•Learning
•Respect AAS program philosophy of a
learning, adaptive program
Making a difference in the lives of the poor

7. 4. Consortium Evaluation Perspectives: A
Performance Management System
Intermediate development outcomes:
•represent a change, in the medium term, which is intended to
affect positively the welfare of the targeted population or
environment resulting, in part, from research carried out by the
CGIAR (and its partners)
•are attributable to CRP-level activities
The CRPs are expected to document these development
outcomes through periodic studies on outcome and impact.
Source: ISPC. Strengthening Strategy and Results Framework through Prioritization. June 2012
Making a difference in the lives of the poor

8. 4. Consortium Evaluation Perspectives:
A Performance Management System
Benefits for the centers are to be able to answer these questions
•How are we performing and what impact are we having in achieving the
SLOs?
•Are we learning and using what we have learned to improve?
•What changes can we make within the CG to be able to provide better
answers to questions 1 and 2?
•CRPs will be held accountable – but be left to implement flexibly
Making a difference in the lives of the poor

9. Impacts - change in problem
- change in opportunities
Development - change in actions/behaviour of stakeholders
Outcomes - change in productivity
- change in equity/empowerment
- change in market conditions
- change in investments
- change in security of assets/habitats
Research - recognition/appreciation of research knowledge
Outcomes - use of knowledge by partners
- mobilisation of new capacity
- extension of technology/materials
- change in policy environment
Outputs - change in knowledge
- change in capacity
- change in technology
- change in materials
- change in policy options
- change in awareness/understanding
Making a difference in the lives of the poor

10. 5. Attributes of NRM programs
Generic attributes of NRM programs
•Multi-stakeholder participation and coordinated action
•Multi-leveled: farm, landscape, region…
•Uncertain and lengthy trajectories for impact
•Knowledge generation
•Contextualized knowledge is vital
•Systems integration
•Operates in areas of limited/little prior or reliable knowledge
•Institutional concerns
•Emerging outcomes
Source: White paper
Making a difference in the lives of the poor

11. 5. Selected attributes of the AAS program
•Strongly place based
•Challenging agro-ecosystems
•Marginalized populations
•Multi-stakeholder participation
•A major process outcome
•Increases in network capital
•Interventions at multiple levels
•Strong commitment to community-level agendas
•Regional and global agendas too
•Multiple-interventions
•‘Research in development’ implies that development is happening
•Overlapping programs and many actors
•Hub-level NRM issues offer opportunities and create constraints
•Emergent effects
•Contextualized causal mechanisms
•Long term processes
Making a difference in the lives of the poor

12. 5. Hub Development Challenges
Malaita, Solomon Islands
•Rural people in the hub face major challenges from rising population and diminishing marine
resources. The development challenge is to improve their lives through more productive, diversified
livelihoods that empower communities to be better able to adapt to change and make more effective
use of their marine resources. The research challenge is to develop and test alternative approaches to
livelihood diversification and resource stewardship that will accelerate development and restore the
productivity of their resources.
Barotse Floodplain, Zambia
•To make effective use of the seasonal flooding and natural resources in the Barotse floodplain system
through more productive and diversified aquatic agricultural management practices and technologies
that improve the lives and livelihoods of the poor.
Khulna Floodplain, Bangladesh
•We seek positive transformational change in the lives and livelihoods of poor AAS-dependent farmers
and their communities, particularly women and youth. We strive for empowered communities that
lead in the innovation and adoption of more productive, diversified and resilient practices and
technologies and demand a more equitable role in the management of natural resources. With this
enhanced capacity and leadership they will make more effective use of water, land and biodiversity
resources, better access information and markets and continually adapt to a dynamic Khulna floodplain
system. Making a difference in the lives of the poor

13. 5. Hub Development Challenges
Malaita, Solomon Islands
•NRM challenge – depleted reef resources, mangrove management
Barotse Floodplain, Zambia
•NRM challenge – seasonal flood management, community canal management,
uplands community forestry management
Khulna Floodplain, Bangladesh
•NRM challenge – community polder management, salt water intrusion, land
subsidence
Making a difference in the lives of the poor

14. 5. AAS Intermediate Development
Outcomes
Time and level slicing – In X AAS Hubs:
•Adoption outcomes –Y Households have adopted AAS technologies and practices
•Sustainable Livelihoods Outcomes –Y communities with enhanced capital to respond to
the hub development challenge (HDC)
•Local institutional change outcomes – an empowerment indicator (e.g. # of orgs utilizing
resilience approaches to NRM challenges)
•Hub-level institutional change – key stakeholders are supporting community action to
tackle HDC
•CG system level institutional change – evidence of adoption of AAS thinking and practice
Making a difference in the lives of the poor

15. 6. Evaluation Questions
•
•Did the intervention make a difference?
•How has the intervention made a difference?
•Will the intervention work elsewhere?
Source: Stern et al. Broadening the Range of Designs and Methods for Impact Evaluations. 2012
To what extent can a specific impact be attributed to the intervention?
Making a difference in the lives of the poor

16. Traditional IA question in CG
Variations of question 1
To what extent can a specific impact be attributed to the intervention?
•What is the net effect the intervention?
•Did the intervention work?
•How much impact can be attributed to the intervention?
•What would have happened without the intervention?
Making a difference in the lives of the poor

17. What is the net effect of the
intervention?
Making a difference in the lives of the poor

18. NRM Evaluation questions
Questions that focus consideration of:
•The continued rationale for the program
•Implementation of the program
•To what extent the program worked (had intended results)
•Why the program works the way it does
•Whether the program will continue to work
Making a difference in the lives of the poor

19. 7. Available Designs
Design approaches
•Experimental
•Statistical
•Theory-based
•Case-based
•Participatory
•Synthesis studies
Source: SternMaking a difference in the lives of the poor
et al. Broadening the Range of Designs and Methods for Impact Evaluations. 2012

20. Table 2: Design, Variants and Causal Inference
Designs Specific Variants Basis for Causal Inference
Experimental RCTs Counterfactuals; the co-presence of cause and
Quasi Experiments, effects
Natural Experiments
Statistical Statistical Modelling Correlation between cause and effect or between
Longitudinal Studies variables, influence of (usually) isolatable
Econometrics multiple causes on a single effect
Control for ‘confounders’
Theory-based Causal process designs: Theory of Change, Process tracing, Identification/confirmation of causal processes
Contribution Analysis, Impact Pathways, or ‘chains’,
Causal mechanism designs: Realist evaluation, Congruence Supporting factors and mechanisms at work in
analysis context
‘Case-based’ Interpretative: Naturalistic, Comparison across and within cases of
Grounded theory, Ethnography combinations of causal factors
Structured: Configurations, QCA, Within-Case- Analysis, Analytic generalisation based on theory
Simulations and network analysis
Participatory Normative designs: Participatory or democratic evaluation, Validation by participants that their actions and
Empowerment evaluation, experienced effects are ‘caused’ by programme
Agency designs: Learning by doing, Policy dialogue, Collaborative Adoption, customisation and commitment to a
Action Research goal
Synthesis studies Meta analysis, Narrative synthesis, Realist based synthesis Accumulation and aggregation within a number
of perspectives (statistical, theory based,
ethnographic etc.)
Making a difference in the lives of the poor