This document discusses theory of change and its importance for evaluation. It begins by introducing theory of change and explaining that it is a process for exploring how change happens in a particular context. It then discusses building a theory of change by defining a program, its outcomes and intermediate steps, and identifying assumptions. The document explains that theory of change is important for evaluators to consider process and for programmers to be results-oriented. It also notes a common criticism is that theory of change can oversimplify programs.
2. Presentation Overview
Introduction to Theory of Change
Building a Theory of Change
Why is Theory of Change Important?
Testing competing Theories of Change
3. Presentation Overview
Introduction to Theory of Change
Building a Theory of Change
Why is Theory of Change Important?
Testing competing Theories of Change
4. Theory of Change (ToC)
Definition
• Theory of change is an on-going process of reflection to explore
change and how it happens – and what that means in a
particular context, sector, and/or group of people.
ToC thinking
• Structured way of thinking about change and impact
organizations would like to achieve
• Integrated approach to programme design, implementation,
M+E, and communication
5. Causal Hypothesis
Q: How do I expect results to be achieved?
A: If [inputs] and [activities] produce [outputs] this should lead to
[outcomes] which will ultimately contribute to [goal].
6. Presentation Overview
Introduction to Theory of Change
Building a Theory of Change
Why is Theory of Change Important?
Testing competing Theories of Change
7. Theory of Change: Main Questions
What is the programme?
What outcomes does the programme aim to achieve?
What intermediate steps lead to those outcomes?
What assumptions are associated with each link in the causal
chain?
How can we measure outcomes?
8. What is the Programme?
Programme Design
Relevance
Target Audience
Social and Political Content
Potential Threats and Challenges
9. Theory of Change: Define the Programme
Incentives for
Immunization
Immunization
Camps
10. I
N
C
R
E
A
S
E
D
I
M
M
U
N
I
Z
A
T
I
O
N
Theory of Change: Define the Outcomes
Incentives for
Immunization
Immunization
Camps
11. I
N
C
R
E
A
S
E
D
I
M
M
U
N
I
Z
A
T
I
O
N
Theory of Change: Intermediate Steps
Incentives for
Immunization Parents bring
children to the
camps
Immunization
Camps
12. I
N
C
R
E
A
S
E
D
I
M
M
U
N
I
Z
A
T
I
O
N
Theory of Change: Assumptions
Incentives for
Immunization Parents bring
children to the
camps
Immunization
Camps Camp
provides
immunizations
Parents value
incentives
Parents trust
camps
Incentives
paid regularly
13. ToC: School-Based Malaria Intervention
Malaria
Intervention
Assumptions
Assumptions
Children
adhere to
full med
regime
Higher
Test
Scores
Test
accurately
measures
knowledge
Increased
School
Attendance
Reduced
Clinical
Attacks
Reduced
Asymptomatic
Parasitemia
Reduced
Anemia
Improved
Cognition
Increased
Concentratio
n
I
N
C
R
E
A
S
E
D
K
N
O
W
L
E
D
G
E
14. Theory of Change Levels
Inputs/
Programme
Activities
Outputs
Intermediate
outcomes
Goal
What we do
as a part of
the
programme -
deliver, teach,
offer loans,
etc.
What are the
resources
used –funds,
staff,
equipment,
curriculum, all
materials.
Tangible
products or
services
produced as
a result of
the activities
- usually can
be counted.
Short-term
behavioral
changes that
result from
the outputs -
preventive
health habits,
usage of
tablets.
Long-term
changes
that result
from
outcomes –
the result of
the
programme.
15. Good indicators
Quantitative and qualitative
Standard of comparison (i.e. baseline v. endline, defining “high-quality,” etc.)
SMART
• Specific - Ask (answer) one question at a time
• Measurable - Quantifiable, accurate, unbiased, sensitive
• Achievable - Is this impact realistic? Are the goals attainable?
• Relevant - Is this the most relevant programme indicator given
the needs
• Time-bound - Has boundaries. When’s the deadline?
16. Good Indicators
Does the definition adequately describe our concept?
Can the variable be easily measured?
Can we collect data to measure the variable?
17. Increased Immunization Coverage
Brainstorm in groups:
• Does the definition adequately describe our concept?
• Can the variable be easily measured?
• Can we collect data to measure the variable?
18. Log Frame
Objectives
Hierarchy
Indicators Sources of
Verification
Assumptions /
Threats
Impact
(Goal/ Overall
objective)
Increased
immunization
Immunization
rates
Household
survey
Adequate
vaccine supply,
parents do not
have second
thoughts
Outcome
(Project
Objective)
Parents
attend the
immunization
camps
repeatedly
Follow-up
attendance
Household
survey;
Immunizatio
n card
Parents have the
time to come
Outputs Immunization
camps are
reliably open;
Incentives are
delivered
Number of kg
bags
delivered;
Camp
schedules
Random
audits; Camp
administrativ
e data
Nurses/assistants
will show up to
camp and give
out incentives
properly
Inputs
(Activities)
Camps +
incentives are
established
Camps are
built,
functional
Random
audits of
camps
Sufficient
materials,
funding,
manpower
Needs
assessment
Impact
evaluation
Process
evaluation
19. What is the best time to create a theory of change?
A. Before commencing an evaluation
B. Before data analysis
C. During the programme design
D. Before data collection
25% 25% 25% 25%
Before data analysis
During the programme ...
Before commencing an ...
Before data collection
20. Presentation Overview
Introduction to Theory of Change
Building a Theory of Change
Why is Theory of Change Important?
Testing competing Theories of Change
22. Solving the Black Box Problem
Low immunization rates Needs
Assessment
Intervention Intervention
design/Inputs
Final
Black Box
No increase in full immunization outcome
23. Identifying Theory Failure vs. Implementation Failure
Successful intervention
Inputs Activities Outputs Outcomes Goal
Implementation failure
Inputs Activities Outputs Outcomes Goal
Theory failure
Inputs Activities Outputs Outcomes Goal
24. Why is Theory of Change Important
For evaluators, reminds us
What is it? Example Components Assumptions Conclusion
to consider process
For programmers, it helps
us be results oriented
25. What is the Main Criticism of Theory of Change?
A. Over-simplify the programme
B. Long-term effects not
considered
C. Unintended consequences not
considered
D. Does not consider that
25% 25% 25% 25%
Over-simplify the prog...
programme to outcomes is not
uni-directional
Unintended consequenc..
Long-term effects not co...
Does not consider that p...
26. Presentation Overview
Introduction to Theory of Change
Building a Theory of Change
Why is Theory of Change Important?
Testing competing Theories of Change
27. Testing Competing Theories of Change
Different disciplines have different theories
• Public Health
• Education
• Economics
• Anthropology
• Sociology
• Political Science
28. Testing Competing Theories of Change
Toilets in
Schools
Children
missing school
due to
waterborne
disease
Children use
toilets
Increased
School
Attendance
Less
waterborne
disease
Girls feel more
comfortable
coming to
school
Female
dropout rate
decreases
Girls missing
school due to
lack of toilets
Girls use toilets
29. Theory of Change: Main Questions
What is the programme?
What outcomes does the programme aim to achieve?
What intermediate steps lead to those outcomes?
What assumptions are associated with each link in the causal
chain?
How can we measure outcomes?
Notas do Editor
Rooted in context and stakeholder analysis - Causal pathway (the arrows) - Identifies assumptions - Derived through a participatory process
Components:
Activity versus outputs: Output is the good or service delivered as a result of the programme; activities support outputs but cannot alone deliver outcomes
Intermediate versus high level outcomes: Example: student knowledge has increased is IO, but a stronger economy because of higher educated workforce is a high level outcome. IO will lead to HLO
I will cover this more tomorrow in the Measurement lecture
Translating the theory of change into a log framework, which many of you are familiar with, to map the change in the project.
The needs assessment underpins everything. Assessment of inputs and outputs – process evaluation, when you are looking at outcomes and impacts (changes and goals), then you switch over to impact evaluation
**Adapted from Marc Shotland, JPAL
So we’ve done our needs assessment. We have designed and implemented our intervention.
But when we come back to evaluate the programme we find that there is no growth in youth employment.
If we didn’t do any process monitoring or process evaluation—in other words, if we didn’t track every step of our theory of change—we might not be able to figure out why the intervention didn’t work, or where it broke down.
If, as evaluators, we look only at our final outcomes or impact, we may not understand why it worked, or if it didn’t, where it broke down.
Let’s peer into the black box. Were our activities implemented as planned? Were outputs produced? Were assumptions met? Did the necessary outcomes change?
In this example one of our major assumptions—which is that youth were motivated to start new businesses—was not met. As a result, even though we implemented our intervention appropriately our goal was not achieved.
This is why it is important to articulate a theory of change in advance and measure each step of the theory of change (through process monitoring and evaluation) as part of an effective impact evaluation.
**Adapted from Weiss, Evaluation, 1998, p 129
The black box problem can be described in other terms.
In a successful intervention that sees the expected outcome and goal change, our theory of change is realized. That is, inputs are sufficient to conduct the activities, the appropriate products and services are provided, our outcomes change in the way we hope and this contributes to our targeted goal.
It is not unusual, however, that we do not see the outcome or goal change that we had hoped for. Without an articulated and measured theory of change, however, we might not know why that is. Is it because our implementation process was problematic? That is, our inputs did were not sufficient/appropriate to implement the planned activities, or somehow the activities we implemented did not lead to the right outputs. This is called implementation failure.
On the other hand, maybe our implementation process was tight. Our resources were sufficient. Our activities were implemented as planned and the services provided were good. Instead, the idea behind our programme was faulty. That is, the products and services we provided didn’t actually lead to the expect knowledge or behavior change, and so our ultimate goal was not achieved. If we have breakdown at this end of the chain (outputsoutcomesimpact) this is called theory failure.
The implications are really different because with implementation failure we can only know that our mechanism didn’t work. With theory failure we know that our idea doesn’t work. If we couldn’t distinguish between implementation failure and theory failure it would be very difficult to use evaluation results to shape policy and programmeming because we wouldn’t know whether it was a bad idea or whether we had just implemented the intervention so poorly that the causal chain broke down.
You all have this theory of change diagram in handout form. On this slide it’s not important that you be able to read what is in the boxes.
The idea here is that we’ve talked quite a bit about what theory of change is. Now why is it important?
For programme people, it helps us keep a focus on results—eye on the prize.
Programme implementation folks are usually focused on activities and outputs: How many trainings have we completed? How many beneficiaries have we reached?
We should move away from interventions for interventions’ sake toward development that explicitly serves a larger purpose and goal.
E.g., In the example above, that means I shouldn’t train partner NGOs just because I think it’s fun, but I should also be oriented toward the reason I am doing it.
For evaluators, it reminds us to consider process
Evaluators are very focused on assessing outcome change. But sometimes this is done with a blind eye to how change was achieved.
We call this the black box problem (next slide).
ASK: How many of you have heard of the black box problem before?
Can anyone take a stab at explaining the black box problem?