Searching for outcomes in rural Tanzania: Harvesting directly from those influenced: Implications for evaluating behavioural change initiatives with communities
The benefits and challenges of using Outcome Harvesting to evaluate a short-term intervention are explored using the example of an 18 month social change project supported by the UK Department for International Development in Tanzania. The project was that was highly ambitious: it sought to influence changes in gender attitudes and behaviour of the general public in Tanzania. Challenges included the lack of outcome indications in project document and the lack of knowledge of outcomes among project personnel. Outcome Harvesting was adapted to allow the harvesting of outcomes using focus groups of those the project sought to influence directly. The concept of ‘proto-outcome’ was used for suggestions of attitude changes that may lead ultimately to behaviour changes. Substantiation of outcomes involved not only third parties but direct observation. The resulting descriptions of outcomes and the evaluation findings proved valuable for learning in the organisation, Search for Common Ground.
Semelhante a Searching for outcomes in rural Tanzania: Harvesting directly from those influenced: Implications for evaluating behavioural change initiatives with communities
Semelhante a Searching for outcomes in rural Tanzania: Harvesting directly from those influenced: Implications for evaluating behavioural change initiatives with communities (20)
Better SAFe than sorry - Why scaled agile frameworks do not necessarily impro...
Searching for outcomes in rural Tanzania: Harvesting directly from those influenced: Implications for evaluating behavioural change initiatives with communities
1. Searching for outcomes in rural Tanzania:
Harvesting directly from those influenced
Implications for evaluating behavioural change
initiatives with communities
John Mauremootoo, Richard Smith, Dunstan Kishekya
Presentation given at the American Evaluation Association on 12th November 2015 in the
session entitled: The strengths and challenges of Outcome Harvesting for Evaluating in
Complex Situations. Experiences from around the World.
3. The Project
The Team Tanzania (The Team) one
of 15 completed or ongoing
projects implemented in Africa and
Asia by Search for Common
Ground (SFCG) using The Team
concept – The Team is a vehicle for
tackling polarizing societal issues
by stimulating learning in a
persuasive, but non-
confrontational manner.
For Tanzania the issue chosen was
gender equality
Using an ‘edutainment’ approach
(TV and radio soap operas) the
project planned to reach and
influence a wide audience.
4. Goal To contribute
to strengthening the
implementation and
enforcement of
gender-sensitive
legislation in
Tanzania by
influencing changes
in gender attitudes
and behaviour of the
general public
11. The Project Theory of Change
Reach
Response
Resonance
Numbers who watch and listen to the show
The attitude and knowledge changes at
the individual and institutional levels
The actions that have been triggered by
the programmes
12. Evaluation Method
Identification, description and interpretation of outcomes through Outcome
Harvesting (using an OM-inspired definition of Outcome as changes in actions,
relationships, policies or practices of one or more social actors influenced by the
intervention).
With especial emphasis on harvesting outcomes of social actors the project had
been seeking to influence by mobile cinema screenings and follow up discussions.
13. Evaluation challenges
The “Blank Slate of:
• Poor understanding of outcome concept among project implementation team
• Little or no knowledge of outcomes among the project implementation team
• No documentation of outcomes in reports
Short duration of assignment given the fact
that all outcome data had to be collected from
scratch
15. Focus Group Discussion
Guiding questions
explained issues such as:
• The definition of
outcomes
• The definition of
contribution
• How to determine how
importance of The
Team’s contribution
• Sources of
substantiation
• Similar outcomes
• Negative outcomes
Data collection via focus group discussion
16. “Treatment” “Control”
Harvesting outcomes through focus
group discussions with social actors
the project had been seeking to
influence by mobile cinema
screenings and follow up discussions
Harvesting outcomes from ‘control
groups’ – target group
representatives who had not
participated in the mobile screenings
and discussions
18. Results: The Team Contribution
54 outcomes and 10 “proto-outcomes”
0 10 20 30 40 50
Useful (25)
Very important (25)
Important (14)
percentage
Strengthened and / or brought forward
changes that may have happened to
some extent anyway
Either essential to the change or greatly
accelerated it
Strengthened and / or brought forward
changes that may have happened to some
extent anyway
Helped realise a change that may not otherwise have happened or
would have happened very much more slowly
27. 1. Outcome Mapping Concepts
OM concepts of outcomes as behavioural change and contribution and the
differences between knowledge, attitude and behaviour were easy and
intuitive to grasp for the participating communities.
28. 2. Knowledge & Attitude Changes
Claimed attitude changes even without behavioural change are important and can be
readily assessed using OH
These “proto-outcomes” helped:
• To understand the claimed process of change even where there have been no
tangible changes in behaviour
• To compare the quality and process of change between “treatment” and “control”
• Inform the change agent
29. 3. Utility of Control Groups
In situations where there is a well-defined theory of change the control
group provides a counter-factual
30. 4. Focus Group Format
The focus group format is effective for discussing less sensitive topics but may not be
the best way of evaluating results concerning sensitive issues and totally inappropriate
for the most sensitive such as rape
31. 5. The value of Follow up
The importance of an iterative follow-up process to enrich the result
description and enhance its credibility.
32. 6. The Timing of the Evaluation
a) It was surprising there were outcomes from a very ambitious project when its 18
months had not yet finished,
b) It was a snap shot & provided credible evidence of change
c) Follow up was needed, e.g. after say 2 years
d) It ideally should really have been a formative evaluation.
33. Now What?
If we consider the
evaluation as an
intervention then both its
findings and the process
need to be built on to help
maximise positive change.
This was not formally done
for a variety of reasons
which in our opinion was a
lost opportunity as is often
the case for evaluations.
34. • Ansila Marandu (evaluation assistant)
• Paul Glick (Search for Common Ground)
• Cornelia Wamba (Search for Common Ground)
• Stella Msami (Search for Common Ground)
• VanessaCorlazzoli (Search for Common Ground)
• The Mvomero Organizations Coalition
• Women and Girls Fight illiteracy and Poverty
Organisation Save the Children Tanzania
• Kate Dyer (AcT and KPMG)
• Layla Ghaid (AcT and KPMG)
Acknowledgements
4 results
Example
Outcome Description
Following The Team Mairo Keria Mugesa has involved his wife in decision-making and joint ownership of their business. Their dairy cow is now under joint ownership, they bought 200 chicks together in July 2013, constructed a well jointly from July 2013 and have managed to build an improved home.
Contribution Description
After The Team (specifically the incidences when the female teacher being ridiculed by male teachers and even male students) and the FGD in which women's right to own property was discussed, Mairo could see the potential socio-economic benefits of involving his wife in decision-‐making and ownership.
7 results
Example
Outcome Description
From June 2013 Tatu Mbonde the leader of the Mvomero widow's group has initiated a participatory planning process for the group to help them to earn income from agriculture and save to help members in trouble.
Contribution Description
Before watching The Team and being involved in the focus group discussions Tatu and her group had no idea of planning together at all. From watching The Team and the discussions that followed, Tatu learned that teamwork among community members brings about efficiency.
6 results
Example
Outcome Description
Following The Team and up to October 2013 Marwa Weibina had stopped beating his wife.
Contribution Description
After The Team, Marwa Mirumbe and his friends and advised Marwa Weibina against beating his wife. This followed the Team episodes and the focus group discussions in which the negative effects of GBV were highlighted. Alternative gentle approaches to reconciliation such as consultations with elders were discussed.
4 results
Example
Outcome Description
In April 2013, Amina Daudi's parents allowed her 14 year old younger sister to attend year one of secondary school.
Contribution Description
Watching the aspect of The Team TV programme of women trying to pull girls from school for marriage and the follow-up discussions on girls’ identity helped Amina to realise that girls have unique opportunities for schooling. This motivated her to fight for her sister's right to attend school.
Other Outcomes (in line with objectives but not clearly related to pre-defined priority themes)
20 results related to contributions to gender equality with financial implications
6 related to participatory approaches to decision making and planning
Financial related
Example
Outcome Description
Following The Team Dhamira Athumani who was jobless young lady began to grow vegetables which she is now selling. She has also joined a theatre and performance art group where she is learning drama.
Contribution Description
Dhamira Athumani was one of the women that Twalah Mbwangali (the 47 year old male Chairperson of Miembe Miwili sub--‐village) counselled to use different strategies to fight poverty. This action was the result of watching The Team and seeing how Miss Wito used different strategies to achieve her goal.
Participatory approaches
Example
Outcome Description
Following The Team Zainabu Said Zuberi, a 33 year old female farmer changed her approach to dealing with conflict in the family and with neighbours to a participatory approach that involves being humble.
Contribution Description
By adopting this approach she was able to help resolve a misunderstanding between her neighbours. Through The Team Zainabu saw Miss Wito’s consultative approach to leadership.
5. Value of focus group discussions for capturing the essence of a result and the importance of an iterative follow-up process to enrich the result description and enhance its credibility.
To be robust, it is arguable that each new piece of information needs triangulating. A tedious process if followed literally and probably only possible for a limited number of outcomes. But if some uncertainty that the final outcome / contribution descriptions are fully accepted by all informants, then it is an enriching and probably practical approach.
Evaluation use:
If we believe the Simultaneity Principle (from Appreciative Inquiry) that inquiry is an intervention then an evaluation such as this has considerable potential as a means for accelerating the change process through evaluation use – both of its findings and the process.
The use of findings is obvious but there is also the use of the process, e.g. the focus group discussions informed others in the group about outcomes and this may have affected their attitudes and subsequent behaviour. The partner organisations became more interested. So we are missing a trick if we do not go on to know, show and grow the bright spots. The evaluator can contribute to this take off process by having a thorough follow-up session with the change agent and partners… sadly this often does not happen for all sorts of reasons.