Presentation about principles, tools, and experience sin measuring the policy influence of research. Developed by RAPID programme at ODI.

1. Measuring policy influence:
like measuring thin air?
John Young: j.young@odi.org.uk
Arnaldo Pellini: a.pellini.ra@odi.org.uk

2. RAPID
Power, Politics and evidence use
Knowledge intermediaries
and interactions Evidence production and
communication

3. Policy processes are...
Cabinet
Donors
Policy
Formula/on
Agenda
Parliament
Se3ng
Decision
Making
Civil
Society
Monitoring
and
Ministries
Evalua/on
Policy
Implementa/on
Private
Sector

4. The Cynefn Framework
4

5. Policy change
• Discursive: Client-
focused services
• Attitudinal: Farmers
have good ideas
• Procedural: Participatory
approaches to service development
• Content: UU20, UU25. New guidelines
• Behavioural: Approach being applied in
practice
5

6. Focus on behaviour change
Project
Team
Other
Actors
Inputs Activities Outputs Outcomes
Outcome
Outcomes Impact
Impact
Impact
Behaviour
6 Change

7. RAPID Outcome Mapping Approach
Define your
policy
objectives

8. RAPID Outcome Mapping Approach
Media strategy
Online
communications
Develop a network
or partnership
Academic research
communications
Policy advocacy
coalition
More research

9. Why do M&E?
• To learn about what works
• To manage better
• To account:
– to donors
– to recipients
9

10. Methods
• Classical case studies (IDRC, IFPRI)
• Episode studies (ODI/RAPID)
• Stories of Change (Denning)
• Micro-Narratives (Snowden)
• Impact matrices (Davies)
• Peer evaluations (CHSRF)
• Systematic reviews?
• RCTs?
10

11. Outcome Mapping
OUTCOME
MAPPING:
Building
Learning and
Reflection into
Development
Programs
Sarah Earl, Fred
Carden, and Terry
Smutylo
http://www.idrc.ca/en/ev-9330-201-1-
DO_TOPIC.html
11

12. Social Network Analysis
12 www.odi.org.uk/RAPID/Tools/Toolkits/KM/Social_network_analysi

13. RAPID Outcome Assessment
13 www.odi.org.uk/RAPID/Publications/RAPID_WP_266.htm

14. A systematic approach
1. Strategy and direction –are you
doing the right thing?
2. Management –are you doing what
you planned to do?
3. Outputs – are the outputs
appropriate for the audience?
4. Uptake – are people aware of your
work?
5. Outcomes and impacts –are you
having any impact?
14

15. Conclusions
Research to influence: Measuring impact
• Clear objectives • Clear objectives
• Theory of change
• Understand the context
• 5-levels
• Theory of change
• Multiple methods
• Iterative / learning • Triangulation
approach
• Expect the unexpected
15

16. Recommendations
• Strategy: theory of change, impact
pathways, peer review, log frame
• Management: process records, appreciative
inquiry, AARs, PRINCE2
• Output: logs, peer-review,
• Uptake: logs, webstats, surveys
• Impact: outcome mapping, stories of
change, episode studies, peer review
16

17. VDR 2010
VDR 2010: assessing policy
influence

18. VDR 2010
VDR 2010: Modern Institutions
• Research output of a large governance
programme (DfID)
• 12 months of research and consultations
• Presented by WB + 13 donors in 12/2009
• Policy influence assessment in 03/2011

19. Confirm the policy change
Attitudinal change
Procedural change
Policy content
Behaviour change

20. Policy influencing approaches
Evidence / science
based
Advising Advocacy
VDR 2010
Inside track / Outside track /
cooperation pressure
Lobbying Activism
Interest / values
based
Start and Hovland
(2004)

21. Areas of M&E
Strategy and direction Logframes, ToC
Project Governance,
Management processes
Research Management
Peer reviews, After
Outputs
Action Reviews
Citation Analysis, Uptake
Uptake
logs, User Surveys, …
Stories of Change,
Outcomes and impacts
Episode Studies, …
Hovland 2007

22. Areas of M&E
Areas of evaluation Planned
Strategy and direction
Management processes
Outputs
Data on uptake of
Uptake VDR 2010
Outcomes and impacts Stories of Change

23. What we did
• Collaborate with a local research/advocacy
organization
• Citation analysis Google search
• Interviews with donors representatives
• Interviews with GoV advisers involved in the
VDR 2010 process
• Interviews with GoV officials

24. What we did/learned
• Cultural norms influence the process
• Citation, Google search, and time
• Uptake and www traffic data
• Management of the process emerged as an
important theme
• Perception of the quality of the report
• Too early to speak of impact

25. What we did/learned
Areas of evaluation Planned What we did/learned
Strategy and Perception about the
direction timing of the VDR 2010
Management Perception of the
processes process
Perception of the
Outputs
quality of the VDR 2010
Data on uptake of Data on uptake of the
Uptake VDR 2010 VDR 2010
Outcomes and Story of Change /
impacts Episode Study

26. What we did/learned
Assessing the policy influence of research: a
case study of governance research in Viet
Nam
ODI Background Notes, May 2012
(http://bit.ly/LNLHE0)
Assessment of the delivery of the
Vietnam development report 2010 (Full
report)
(http://bit.ly/LNOJZ1)

27. M&E in ODI
• ODI has 12 core research programs
• 145 publications / 74 blogs / 13 issues
Development Policy Review and Disasters
(2009-2010)
• 20 public events / 4500 participants
(2009-2010)
• Advisory work

28. M&E in ODI
Strategy and ODI Strategy / Program Strategy / Reviews by
direction SMT / ToCs
Institute-wide management processes / staff
Management surveys / project completion reports / after
processes action reviews / internal peer review
Peer reviews / quality assurance / communication
Outputs team input / user surveys / end of project report
+ after action reviews

29. M&E in ODI
Note by communication team /
Uptake CommStat / regular updates on
uptake / uptake log

30. M&E in ODI
CommStat ODI RAPID July – September 2011

31. M&E in ODI

32. M&E in ODI
Outcome / Stories of Change page / Review by TT
Impact heads (2008) / Program evaluations

33. M&E in ODI
http://www.odi.org.uk/work/stories-change/

34. Thank you