Presentation on CCAFS's experience with theories of change, results-based management, and monitoring, evaluation and learning.

1. Evidence On Trial: weighing the value of
evidence in academic enquiry, policy and
everyday life
CCAFS outcomes and targets
Tonya Schuetz, Philip Thornton, Wiebke Foerch
Durham 11 July 2016

2. Structure
CCAFS Experience
Background
Theories of Changes
RBM MEL Framework
Issue of Evidence
Credibility of Outcomes
Operational Support
Challenges & opportunities

3. CGIAR,
a global partnership that unites organizations
engaged in research for a food secure future
Background
FTA MAIZE Fish WHEAT PIM A4NH
RICE Livestock RTB DCLAS CCAFS WLE

4. Strategy and Results
Framework 2017-30
Outcome Indicators
and Targets
Background

5. 4. Climate information services
and climate-informed safety
nets
1. Policies and institutions for
climate-resilient food systems
2. Climate smart
technologies,
practices, and
portfolios
Background - Flagship Programs

6. • Large bodies of work (e.g. US$1.5 million)
• Source of funds: any kind
• Single or Multiple Centers/Partners
• Single or Multiple Regions or Global
• Key feature: fits into an impact pathway
Projects
Background – Regional Programs

7. Theory of Change & Impact Pathways – why?
• Bring in focus on people +
outcomes, i.e. behavioral changes,
+ unanticipated changes +
incentives for change
• Ensure that CCAFS plan of work is
targeted at outcomes, requires ‘use
of outputs’ tasks built into each
activity plan
• Strategically encourage
communication and collaboration,
and improved work plans through
harmonizing IPs
• Revisit trajectory of CCAFS
contributions to change and use
them as foresight/ ex-ante impact
assessment – measure progress
towards change, adapted according
to learning

8. From output-focused supply-driven
(logframe approach) to outcome-
focused demand driven Theories of
Change over a two-year process (starting
from the regions):
• Harmonization achieved through
capacity building (beyond trainings: meetings
and frequent virtual communication) in all
flagships, regions and partners
• Built with partners & projects
• RBM first trialed on one Flagship
• Iterative planning process
for the CRP project portfolio
building
Creating our Theories of Change
• Set global flagships
• Regional priorities
• Calls for concepts
• Regional planning meetings
• Detailed project planning

9. • CCAFS operates across multiple sectors, scales, stakeholders
• Allowing for:
Structured linear thinking
and planning with
assumptions for how desired
changes can happen
Complex systems thinking:
flexibility to react according
to lessons and opportunities
arising during implementation
Research Development
Universities,
research
institutes,
bio-medical
facilities,
genetics etc.
UN,
development
NGOs,
philanthropic
foundations,
governments
Solutions
R4D: How were we supposed to do it?
R4D: What it looks like in practice:
Action
Research
Development
Leadership
Learning
Communications
Partnerships
Trust
Combination of ordered & complex systems

10. MEL theoretical framework
Results-based management
• Accountability for
outcomes: logical chain
• Emphasis on systematic, constructive looped learning from past
experiences and subsequent adaption -adaptive management
• Three thirds principle
engaging with partners to decide what needs to be done and how;
doing the actual research, often in partnership;
sharing results in appropriate formats and strengthening capacity of next users to
utilize the research to achieve outcomes and impact.

11. Examples
• Desk studies (e.g. current World Bank investment on
Climate Smart Agriculture)
• Third-party surveys (deforestation rates in CCAFS
regions)
• Other CGIAR/ CRP surveys (e.g. Climate Smart
Agriculture technologies adopted in CCAFS sites)
• Formal impact assessment and evaluation methods
incl. the ones conducted by independent units
• Project activities (CSA technologies adopted in other
sites, GHG emissions in a landscape, …)
• Other (emails, internal government documents, …)
What kind of “evidence” is needed?
Atproject,flagship,andregionallevel
Changesthroughtime–withbaselines
A lot of methodological development is needed to
make evaluation incl. Impact Assessments to be
adequate for TOC / IP approaches.

12. • Definition of an outcome: ‘use of the research by non-
research partners to develop new, or change, policies and
practices’, i.e. beyond new research designs, dissemination,
training and need to be 1000’ of farmers not involved in the
research
• Evaluation of ‘outcome case studies’
By Whom? 1/3 Internal (2) from Program Management Unit
(eliminated conflict of interest), 1/3 external (1) with experience
in a farmer organization, and 1/3 external (1) with >20 years in
R4D
Criteria for scoring: three criteria with different weights,
(a) quality/clarity of write up - 15%, (b) evidence - 20%,
and (c) significance of outcome - 65%.
Credibility of Outcomes

13. How to strengthen the Outcomes?
• Resubmission of outcomes from previous years shows how
these improve over time due to further scaling.
• Trend over four years (2012-15) of reported outcomes:
• improved quality & relevance for 2015, large jump in no.
• increase in % evaluated as ‘not yet outcomes’
• satisfactory no. of very good outcomes reported for 2015
• it would seem that a lot of valuable time went into
developing Outcome Case Studies which did not yet
meet the outcome criteria.
• seems worthwhile to remind of the criteria.

14. Contribution to some System Level Outcomes, which link with SDGs
CRP defined outcome indicators:
• for the flagship level 2030 and 2022
• Quantitative capturing of outcome target values combined with
• Qualitative narrative descriptions
What is CCAFS being held responsible for?
2 + 3
15equitable national/subnational food system
policies enacted that take into consideration climate
smart practices and strategies, informed using
knowledge, tools and approaches derived from CCAFS
science
1 + 2
10regional/global organisations inform their
equitable institutional investments in climate smart
food systems using CCAFS outputs.
20equitable
national/subnational
jurisdictions will have
increased institutional
investments in climate
smart food systems
2015 2016 2022 2030
Flagship 4: Policies and institutions for climate-resilient food systems

15. Understand ToC
Meaningful performance
expectations for results
(outputs, outcomes)
Measure results, assess
contribution of program to
observed outcomes
Report on performance
achieved against expectations
Practical mechanisms and
tools to ensure balanced
quantitative and qualitative
monitoring
A modular MEL system
CCAFS M&E
System
Modules
Harmoniza-
tion of IPs &
ToCs
Indicators &
Baselines
Reflective
Spaces and
Activities
Reporting
Assessment
& Bonus
allocation
Research
on
Institutional
Transfor-
mation
Process

16. • Planning Work & $$
• System/ donor Reports
• Synthesis at
Flagship level
• Synthesis at
Regional level
• Project Synthesis
• Project evaluation
• Deliverable ranking
• Project annual reporting
• Project annual planning
Program
Mgmt.
Unit
Flagship
Program
Leaders
Regional
Program
Leaders
Project
Leaders
Project SettingProject Setting PlanningPlanning ReportingReporting Synthesis & EvaluationsSynthesis & Evaluations
Process supported by an Online Platform
Iterative processes and built-in
• Looped learning
• System for adaptive management
• Project evaluation (traffic light)
• CRP mapped to outcome targets
• Modular MEL
Project Leaders
Wider system
CRP Managers
CCAFS M&E
System
Modules
Harmoniz
ation of
IPs &
ToCs
Indicators
&
Baselines
Reflective
Spaces
and
Activities
Reporting
Assessm
ent &
Bonus
allocation
Research
on
Institution
al
Transfor-
mation

17. Dynamic program and project,
planning and reporting modules
CoA
Technical features
• Open source
• Interoperable with
or linkage to other
data platforms
• Modular

18. MEL Support Pack
• To support strategy implementation through online platform
for different users

19. Incentives – readiness of a system
• The current CGIAR system’s incentive framework can be
improved to make R4D via a TOC/IP approach work
properly, including its financial system.
• A portfolio approach: in which some projects revolve much
more around science and others around engagement. Rarely will
projects do both. Key is that science and engagement are
happening within a regional or global conceptualized and
coordinated programmatic manner.
• Appropriate performance assessment required:
Evaluation needs to be consistent with different measures
required.
Challenges & Opportunities

20. • Complexity and harmonization – of different levels/scales involved:
System, Program, project, countries
• System level – operationalization of the Strategy and Results
Framework with a System MEL Framework and Monitoring Plan
• Donor demands – outcome delivery promise aspirational vs. R4D
reality
• Balancing act of quantitative and qualitative measuring of outcomes
• Results-based management for R4D - learning from the
development sector – yet there are no off-the-shelf solutions
• Accommodating negative outcomes and asking for those explicitly
• ....
Remaining Challenges

21. The “Evidencers”
(the results agenda)
Vs
“The Complexers”
(complex systems,
emergent
properties, messy)
Oxfam blog, “From Poverty to Power”

22. Key reference documents
• Schuetz, T, Förch, W, Thornton, P, Vasileiou, I. (accepted). Pathway to
Impact: Supporting and evaluating enabling environments for research for
development in Juha I. Uitto, Jyotsna Puri and Rob D. van den Berg, eds.
Evaluating Climate Change for Sustainable Development. Springer:
Dordrecht. Forthcoming in 2016.
• CCAFS Planning and Reporting online Platform, Learning Brief No. 16,
Nov. 2015
• CCAFS Reporting and Evaluation in a results-based management
framework, Learning Brief No.15, reporting cycle, Jul. 2015
• Report to CO on RBM Trial, planning cycle, Dec. 2014
• Lessons and Insights from the CCAFS Results-Based Management Trial,
RBM projects feedback/ experience, survey summary, Dec. 2014
• Lessons in theory of change from a series of regional workshops,
Learning Brief No. 11, Dec. 2014
• CCAFS Theory of Change Facilitation Guide, Dec. 2014
• CCAFS RBM MEL strategy, framework from Jul. 2014