1. Monitoring & Evaluation
IN PRACTICE
TTI PEC Anglophone Africa Regional Peer Learning Workshop
Coconut Grove Regency Hotel | Accra, Ghana
Wednesday, July 16th, 2014
!
!
Mo Adefeso-OlatejuRebecca Oluwatoyin Doherty
2. Linking Monitoring,
Evaluation, & Measurement
SHORT VIDEOS
Importance of monitoring & evaluation
https://www.youtube.com/watch?v=2mBcb-AQ8xQ
4:00minutes - end
!
Indicators
https://www.youtube.com/watch?v=SZqOKW2_jNU
!
Indicators: Basic Concepts
https://www.youtube.com/watch?v=qmrDNuJzRv4
4. 4 BASIC M&E POINTS
MUST BE FOLLOWED
Who is our population? What data is to be collected
from them? Or about them?
How will the data captured enhance communications?
And also fit within the rest of the M&E analysis?
How will the data be analyzed? Who will be doing the
analysis? What do we know about our baseline data?
How will the process be managed? And by whom?
An organizational assessment is needed…in order to
establish a process for M&E!
Source: https://sites.google.com/site/newmediacommunications2010/mobiles-and-m-e/2-case-studies
5. ASSESSMENT OF READINESS
In order for the M&E Function to
succeed, it is necessary to meet two major
conditions:
!
▪ Organizational readiness
– Is there a clear M&E framework with
targets?
– Is the planning process evidence-based?
– Is the TT focused on achieving results
(leadership, communications)
!
▪ Technical readiness
- Do the staff have the relevant experience
and skills?
WORKSHEET
6. What do we monitor?
Inputs. Activities. RESULTS.
WORKSHEET
7. Results chain
Resources Processes
Within control of the organization Partial Org. Control No Org. Control
Sphere of Control: Advocacy, Tactics,
Inputs, Activities, Outputs
Sphere of Influence:
Changes in behavior,
relationships,
attitudes, agenda
Sphere of
Concern: Policy
& Political
Context
WORKSHEET
WORKSHEET
10. Strategies & approaches for identifying
evidence of change
• Signposts
– output log, web analytics, media tracking, citation
tracking
– discourse analysis, social network analysis
• Outcome (or influence) log
– record of interactions with decision-makers, anecdotes
shared by stakeholders (e.g. with the director)
• Feedback mechanisms
– stakeholder surveys
– interviews
– focus group discussions
WORKSHEET
11. Integrated Learning Processes in M&E - Illustration
jan marfeb apr sep novoct decmay julyjune aug
Quarterly
report
Half-yearly & yearly
report
WORKSHEET
14. “If you do what you always did,
you will get what you always got.”
Kenneth W. Jenkins
President, Yonkers NY NAACP
TO THINK OR NOT TO THINK ABOUT M&E?
FOOD FOR THOUGHT…
15. The first step in performance accountability is to
DRAW A FENCE
Around something that has
ORGANIZATIONAL OR FUNCTIONAL IDENTITY
The Whole
Organization
Division A
Division B
Unit
Division C
Function
Unit 1
WORKSHEET
16. HOW DO WE THINK ABOUT
WHO, WHAT & WHY WE
MEASURE?
17. Different Kinds of Progress
1. Data
a. Population indicators Actual turned curves:
movement for the better away from the baseline.
b. Program performance measures:
customer progress and better service:
How much did we do?
How well did we do it?
Is anyone better off?
2. Accomplishments: Positive activities, not included above.
3. Stories behind the statistics that show how individuals are
better off.
18. What’s Next?
A Basic Action Plan for Results Accountability
TRACK 1: POPULATION ACCOUNTABILITY
● Establish results
● Establish indicators, baselines and charts on the wall
● Create an indicators report card
● Set tables (action groups) to turn curves
TRACK 2: PERFORMANCE ACCOUNTABILITY
● Performance measures, and charts on the wall
for programs, agencies and service systems
● Use 7 Questions supervisor by supervisor and program by
program in management, budgeting and strategic planning
20. If I can…. Measure, Track, Improve & Show Performance
Education Performance Accountability
What’s in it for me?
Then…. here’s what’s in it for me:
! Improvement = more money
! Success at the State’s Accountability Game
! Student and Parent Satisfaction
! Tool vs. fingers crossed
! Knowing where your students are between
achievement tests
! Knowing that you’ve made a difference
24. Turn the Curve Exercise: Program Performance
5 min: Starting Points
- timekeeper and reporter
- identify a program to work on
- two hats (yours plus partner’s)
10 min: Performance measure baseline
- choose 1 measure to work on – from the lower right quadrant
- forecast – OK or not OK?
15 min: Story behind the baseline
- causes/forces at work
- information & research agenda part 1 - causes
15 min: What works? (What would it take?)
- what could work to do better?
- each partners contribution
- no-cost / low-cost ideas
- information & research agenda part 2 – what works
10 min: Report Convert notes to one page
Two pointers
to action
WORKSHEET
25. Program: _______________
Performance Measure
(Lay definition)
Performance
Measure
Baseline
Story behind the baseline
---------------------------
--------------------------- (List as many as needed)
Partners
---------------------------
--------------------------- (List as many as needed)
Three Best Ideas – What Works
1. ---------------------------
2. ---------------------------
3. ---------No-cost / low-cost
ONE PAGE Turn the Curve Report: Performance
Sharp
Edges
4. --------- Off the Wall
4. --------- Off the Wall
WORKSHEET
26. 2 customers
1 measure
1 measure
Discuss how
we’re doing
2 partners
2 what works
ideas (1 nc/lc)
Discuss how
to implement
20 Minute
Exercise
40
WORKSHEET
27. 1 population
1 result
1 indicator
Discuss how
we’re doing
2 partners
2 what works
ideas (1 nc/lc)
Discuss how
to implement
20 Minute
Exercise
1. What population are we concerned about?
(e.g. U.S. population)
2. What condition(s) do we want for this
population? (Americans are free of cancer.)
3. How could we measure these conditions? (
(e.g. cancer rates)
4. How are we doing on the most important
of these measures? (baseline history)
5. Who are the partners with a role to play
in doing better?
6. What works – what would it take – to do
better? (What is our role?)
7. What do we propose to do?
POPULATION ACCOUNTABILITY
WORKSHEET
28. Results Based Grantmaking Exercise
A. Result _________________________
B. Indicators
1. _______________
2. _______________
3. _______________
For each:
Getting Better Getting Worse
or About the Same?
C. Story behind the baselines (Why better or worse?)
__________ __________
__________ __________
D. Partners with a role to play
__________ __________
__________ __________
E. What would it take? (5 most important strategies)
1. _____________ 4. ______________
2. _____________ 5. ______________
3. _____________
F. Our Roles (3 best ideas)
1. __________________ 3. _________________
2. __________________ Off the wall:. _________
10 min:
10 min:
10 min:
15 min:
10 min:
WORKSHEET
30. ● # of people served
● % participants who got jobs
● staff turnover rate
● # participants who got jobs
●% of children reading at grade level
● cost per unit of service
● # applications processed
● % patients who fully recover
What Kind of PERFORMANCE MEASURE?
Upper Left
Lower Right
Upper Right
Lower Left
Lower Right
Upper Left
Upper Left
Lower Right
QUANTITY
!
!
How much did
we do?
QUALITY
!
!
How well did
we do?
E
F
O
R
T
E
F
E
C
T
Is anyone better off?
WORKSHEET
31. How much did we do?
Professional Development
How well did we do it?
Is anyone better off?
Quantity Quality
EffectEffort
Customers
# teachers served
# students
# parents
Activities
# training sessions
# hours of training
# study groups
# TA contacts
# mentoring relationships
Common measures
% teachers certified in subjects they teach
% customers satisfied with how service is
delivered
% staff with good morale
Unit cost per training unit (HL #3)
Activity specific measures
% positive evaluations on training
% attendance in study groups
% of faculty participating in study groups
% mentoring pairs who complete mo. rept.
% mentees satisfied with match
#
#
!
#
#
!
#
!
#
#
#
% teachers satisfied that prof dev helped
them be a better teacher (DDA #2)
% teacher retention rate
- total
- “quality” teachers (DDA #1)
% mentees who report relationship
helped build their skills
% teachers who demonstrate new skills
- by observation (HL #2)
- by self report
% students in “trained” classrooms with
improved test scores (HL #1)
% students with reduced behavior
problems
% students with good school attendance
% teachers who report better use of time
SAMPLE
WORKSHEET
32. How much did we do?
Professional Development
How well did we do it?
Is anyone better off?
Quantity Quality
EffectEffort Number of students &
teachers served
!
Number of workshops
Unit cost
!
% of teachers with PD
plan
# teachers who used training (by
self rept or observation)
!
# Teachers retained
!
# Achievement in classroom with
trained teachers
% teachers who used training (by
self rept or observation)
Teacher retention rate
!
Achievement rates in classroom
with trained teachers
33. How much did we do?
Environment: Water Quality
How well did we do it?
Is anyone better off?
Quantity Quality
EffectEffort
Number of
stream stations
monitored
Average sites
per monitor
per month
Number miles
of healthy
streams
Percent miles
of healthy
streams
34. How much did we do?
Commerce Tourism
How well did we do it?
Is anyone better off?
Quantity Quality
EffectEffort Number of
books sent to
interested
consumers
Cost per
inquiry
delivered
Number of
tourists
Number tourist
businesses
Tourism
market share
Growth in
tourist industry
35. How much did we do?
Personnel Department
How well did we do it?
Is anyone better off?
Quantity Quality
EffectEffort
Number of
Applicants
Processed
Average
Recruitment
Period
Workforce
New Hires
Workforce
Turnover Rate
(non-promotions)
Customer
Satisfaction
36. How much did we do?
Every Program
(Financial Performance Measures)
How well did we do it?
Is anyone better off?
Quantity Quality
EffectEffort
Cost
(total)
Unit
Cost
Benefit Value
(total)
Cost-Benefit
(ratio)
37. How much did we do?
Corrections
How well did we do it?
Is anyone better off?
Quantity Quality
EffectEffort
# Inmates Rate of
overcrowding
#
Recidivism
%
Recidivism
38. How much did we do?
Blank
How well did we do it?
Is anyone better off?
Quantity Quality
EffectEffort
X X
X X
41. More Resources
M&E Online Tools
!
ALT METRICS
http://www.researchtoaction.org/2013/09/how-to-harness-the-power-of-altmetrics-a-researchers-guide/
http://www.researchtoaction.org/2013/08/altmetrics-and-the-global-south-increasing-research-visibility/
!
USING GOOGLE TO TRACK AND IMPROVE YOUR RESEARCH IMPACT
http://www.researchtoaction.org/2013/08/using-google-to-track-and-improve-your-research-impact/
http://opendocs.ids.ac.uk/opendocs/handle/123456789/2878#.U8VcXl7yDwJ
!
GOOGLE ANALYTICS
http://www.researchtoaction.org/2013/11/key-indicators-for-google-analytics/
http://www.google.com/analytics/
!
HOW TO GET STARTED WITH GOOGLE ANALYTICS
http://mashable.com/2011/05/23/how-to-use-google-analytics/
!
M&E in Theory and Action
!
MONITORING & EVALUATION: SOME TOOLS, METHODS & APPROACHES
http://gametlibrary.worldbank.org/FILES/215_M&E%20Guidelines%20on%20Indicators%20and%20Data%20Sources.pdf
!
THEORY OF CHANGE VS. LOG FRAMES
http://www.researchtoaction.org/2012/05/theory-of-change-vs-logframes/
http://www.knowledgebrokersforum.org/wiki/item/theory-of-change-vs-logframes
http://mande4mfi.wordpress.com/category/theory-of-change/
!
CASE STUDIES ON M&E
https://sites.google.com/site/newmediacommunications2010/mobiles-and-m-e/2-case-studies
!
EXAMPLES OF PROGRAM OUTCOMES
www.whatworks.org
EVALUATION PLANS
http://nonprofitanswerguide.org/faq/evaluation/what-is-an-evaluation-plan-and-why-create-one/
43. For further inquiries, please
contact:
Dr. Modupe Adefeso-Olateju, mo.olateju@tepcentre.com
Rebecca O. Doherty, doherty.rebecca@gmail.com
THANK YOU