1. Measuring and Improving the Effectiveness
of R&D Systems in Sub-Saharan Africa
Leonard Oruko
IAAE Symposium on Improving Returns to
Agricultural Research in Sub-Saharan Africa
Foz do Iguaçu | 20 August 2012

2. R&D Results Measurement Challenges
• What is an effective R&D system?
– Outputs impact on poverty, food security and income growth
– Generates relevant products and services in a timely fashion
– Has adequate human capacity and financial resources
• Can we demonstrate Results of Ag. R&D Investments?
– Human capital, infrastructure and operational funding constraints
– Long time lags from the point of investment to the manifestation of
returns
– “ A fishing expedition or a shooting range”
– Well functioning support services and institutions must be in place for
research outputs to have an impact on development outcomes

3. R&D Results Measurement Challenges..
• Research evaluation has supported the case for R&D
– The tools and information developed for evidence-based policy were
linked to the development imperatives of the day
• Research evaluation responded to questions being asked
– Economic returns , welfare analysis, priority setting, funding of
research issues
– Concerns with poverty (well beyond producer and consumer surplus),
NRM and sustainability (beyond production systems) , and later
climate change at increasing scale
• Impact assessment has had to balance the needs for
accountability to funders versus learning and change by actors
– Economic return, experiments and quasi experiments (quantitative)
– Utilization-focused evaluation, qualitative

4. Status of Results Measurement in SSA Ag. R&D
Institutions
• Ex-ante impact evaluation
*CAADP Framework
– Evidence of some NARS adopting objective criteria, with support from
the CG Centres
– ASARECA, CORAF and CCARDESA focus on estimating spillover
potential
• Managing research implementation process
– Accountability focus pushing NARS towards RBM-PABRA
– FARA and SRO gravitating around RBM derived CPMF
– CG-CRP
• Ex post impact evaluation
– Adoption
– Precise measurement of impact with RCT emerging as “ the Gold
Standard”

5. Impact of Ag. R&D: Common practice
• Primary focus has been that of establishing the impact on development
outcomes
Source: Block, 2010

6. Outcomes of Ag. R&D
• Estimates of adoption primarily case specific
– Targeted studies estimate adoption levels and determinants of
adoption; guide research planning and priority setting
• Important lessons for R&D results measurement systems
– LSMS-ISA
– DIVA Initiative
Adoption of improved varieties
Crop % cropped area
Maize (in west and central Africa) 67
Cassava 39
Beans 32
Sorghum 14
Source: Alene, et al, 2011

7. Inputs and outputs
• The DIVA Initiative
• Research expenditure
• Full Time Equivalent Scientists(FTEs)
• Research Intensity
• Mean Incidence of varietal Output
Changes in researcher Intensity ratio over time
Commodity 2010 1998
Rice 10.9 6
Maize (west and central Africa) 9.2 10
Cassava 1.2 3
Sorghum 1.7 5
*Beans 33.7 21
Source: Alene, et al,2011

8. Improving the Results measurement of R&D
systems
“A major knowledge gap in understanding and strengthening
R&D systems stems from the lack of empirical application of
framework, metrics, and benchmarks to measure organizational
performance and institutional impact in the context of
agricultural research”
Ragasa, 2011
• Organization design theory in the context of innovation
system
– Coordination mechanisms that provide incentives for innovation
– Demand responsiveness and connectivity to other actors in the
innovation system

9. Empirics from Ghana and Nigeria : Perception
Ratings)
• Output, outcome and impact indicators are standard
-Technologies generated
-Publications
– Adoption of technologies (most researchers unaware of the adoption
rates)
– Limited complementarity and consistency across the indicators
• Connectivity
– Linkage with other researchers exist, limited in the case of extension,
farmers and other innovation actors
• Organization culture and job satisfaction
– Satisfaction with outputs
– Staff morale
– Perception on effectiveness of the organization

10. Way Forwards for results Measurement
• There is no substitute for valid and credible data
– Real time data for operational management not available in the majority of
cases
– Measurement error arising from reported area and output data (LSMS-ISA)
– Data is a valuable resourcetreasure often kept in “armory”
– Challenges with data sharing protocols hence despite the noble intentions
espoused in; CAADP, CRP, SRO
• Getting adequate data for results measurement is costly
– Owing to scarcity of resources , collection of performance data and
information is rarely given priority-donors are pushing for a reversal!
– Greater chances of getting resources when framed as a research endeavor
– Operational management data is often treated as confidential

11. Way Forwards for Results Measurement
• Strategic Focus in SSA
– ASTI initiative to support the NARS in the institutionalization of
data collection and expand to include output indicators
– Work with SRO and RECS
– FARA and SRO’s to focus on quantifying the
externalities/spillovers and, support the NARS in developing
measurement approaches for effective coordination and
management
– NARS to plug into the broader innovation system and NIMES in
order to demonstrate contribution to broader development
agenda

12. Way Forwards for Results Measurement
• Rigorous impact evaluation approaches recommended
– Selective use of RCT and other quasi experimental methods given the
associated costs
– Develop rapid and robust approaches for measuring the impact of
R&D on development outcomes
• Operational management support
– Great research opportunity in the area of agricultural innovation
systems employing management science tools
• Effective results measurement systems respond to
information needs in a timely fashion-an art
– Proactive strategic analyses
– Consistent data collection effort
– Focus on generating evidence and catalyzing use
*Duplication of efforts arising from fragmented approach to data
collection

13. THANK YOU