The document provides an overview of understanding, measuring, and communicating impact. It discusses key concepts like impact, impact measurement, and intended and unintended outcomes. It also covers developing a theory of change to link activities to outcomes, assessing the appropriate strength of evidence needed, and using both qualitative and quantitative tools to measure impact. Common tools include surveys, scales, interviews and case studies. The document emphasizes selecting tools based on outcomes, resources, and evidence needs. It also stresses the importance of clearly articulating impact to different audiences through addressing what problem is being tackled and the evidence of what is being achieved.

1. v
UNDERSTANDING, MEASURING AND COMMUNICATING
YOUR IMPACT
Eibhlin Ni Ogain
Bristol, 21st June, 2013

2. AGENDA
Concepts and definitions
Understanding your impact
and what to measure
Assessing the strength of
evidence you need
Articulating your impact

3. NPC: TRANSFORMING THE CHARITY SECTOR
3
NPC works at the
nexus between
charities and
funders
Charity
SectorFunder
Increasing the
impact of charities
eg, impact-focused
theories of change
Strengthening the
partnership
Eg, collaboration
towards shared
goals
Increasing the
impact of funders
eg, effective
commissioning
Consultancy
Think tank

4. WHAT IS IMPACT MEASUREMENT ANYWAY?
4 4
• What is impact?
– The overall difference an organisation,
programme or intervention makes
• What is impact measurement?
– Planning, managing, tracking and reviewing
how much difference you are making

5. IMPACT: POSITIVE, NEGATIVE, INTENDED,
UNINTENDED COMPONENTS
5
Positive Negative
Intended
School programme that
improves educational
attainment
Campaigns against
multi-national
corporations hurts
shareholder profits
Unintended
School programme improves
parents’ integration in UK
Welfare payments
create poverty traps /
dependency

6. UNDERSTANDING YOUR
IMPACT

7. NPC – Measuring your impact.: Introductory Workshop for Charities 7
7
Tracker 1 Tracker 2 Tracker 3 Tracker 4
WHERE TO BEGIN
Why are we measuring?
Alice: "Would you tell me, please, which way
I ought to go from here?"
Cheshire Cat: "That depends a good deal on
where you want to get to."
Alice: "I don’t much care where."
Cheshire Cat: "Then it doesn’t matter which
way you go."

8. 8
Enable people with mental illness or epilepsy to
live and work successfully in their own communities.”
“Make sure the best of the past is kept to enrich
our lives today and in the future.”
“Relieve distress, mobilise personal resources and
facilitate growth in adolescents towards responsibility and
self-fulfilment.”
“…to help youth and their families to live, work, and study
with dignity, hope, and joy.”
“Transform the charity sector”
MISSIONS

9. COMPONENTS OF NPC’S APPROACH TO
MEASURING SOCIAL IMPACT
9
Strategic vision / goals A well-developed
Theory of
Change
Existing evidence Appropriate measurement tools

10. 1010
HOW TO LINK ACTIVITIES WITH LONG TERM
VISION
101010
• NPC approach: Use theory of change to identify
specifically what a charity hopes to achieve through its
activities, and come up with appropriate measures
• Takes more time than listing measures
• Common approach: list desired outcomes and put
appropriate measures against them
• Ignores distinction between a charity’s direct and indirect
influence
• Risks being a wish list that a charity finds too hard to
achieve

11. THEORY OF CHANGE
• Links activities intermediate outcomes  final outcomes
– clarifies what the activities aim to achieve and how
– provides a structure for identifying what can be measured
– provides the case for why achieving intermediate outcomes is
important
11
A conceptual map of how activities lead to outcomes

12. CASE STUDY
Scenario
• Intensive drama
programme with
young people.
• Programme lasts a
year and frequency
and duration of
contact is high.
• New and untested
approach but
charity believes it
to be very effective.
• Want to generate
evidence to
illustrate this
effectiveness and
expand the
programme to
other locations.
Young
people take
part in drama
workshops
focusing on
role play
Young people have
improved
communication skills
Young people have
more confidence to
seek help
Young people
have improved
behaviour
Young people learn
about different’
viewpoints
Improved
communication with
caregivers and
teachers
Young people get
the help they need
Young people learn
a new skill
Young people feel
better about
themselves
Programme using drama and theatre to improve communication skills and
behaviour of at risk youth

13. TIPS FOR DEVELOPING A THEORY OF
CHANGE
13
• Get the views of lots of people—do you have the right people in your
workshop?
• Do intermediate and final outcomes reflect user experience?
• End goal—How will you know if it has been achieved?
• Backwards mapping—what has to happen for this to happen?
• Outcomes—are they actually outcomes? Are they all on there?
• Assumptions—are your causal links based on evidence or not?
• Enabling factors—what would derail your intervention?

14. COMMON PITFALLS
1. Including non-outcomes
2. Overcomplicating models
3. Over-claiming outcomes
4. Not following a logical flow
5. More than one outcome in the same box
6. Similar outcomes occur more than once
7. Outcomes not specific enough
14

15. ASSESSING THE
STRENGTH OF EVIDENCE
YOU NEED

16. HOW ROBUST DOES EVIDENCE OF YOUR
IMPACT NEED TO BE?
• What do your target
stakeholders (funders?) think?
• What is possible, given
resources etc?
16

17. LEVELS OF EVIDENCE
17
Randomised
control trial
Anecdotes /
quotes
Before and
after survey
Self-reported
change
Case
studies
Control
groups
Credibility
Basic Advanced
Nesta level 1 Nesta level 2 Nesta levels 3, 4, 5

18. LEVELS OF EVIDENCE: EXAMPLE
18
The Brandon Centre provides
counselling and psychotherapy to
young people between the ages of 12
and 21
The Brandon Centre provides counselling and psychotherapy to young people
between the ages of 12 and 21
Level two
• The centre routinely carried before and after assessments
Moving to level five
• Results showed poor outcomes and early drop out for young people with
behavioural problems
• Reviewed evidence on effective interventions and found MST
Needed to show that:
• MST is more effective than current YOT services at reducing behavioural
problems and youth offending

19. LEVELS OF EVIDENCE: EXAMPLE
19
The Brandon Centre provides
counselling and psychotherapy to
young people between the ages of 12
and 21
Level five
• A randomised control trial was needed to answer these questions
• Recruited young offenders on a referral or supervision order
• Young people randomly allocated to two groups: MST or usual YOT services
Young
people
YOT
usual
service
MST
Random
allocation

20. 20
• Before and after assessments and young person’s offending history recorded
• At two year follow up, MST group showed significantly greater reduction in
offending and problem behaviours
• Expansion of MST to other LA’s as a commissioned service
LEVELS OF EVIDENCE: EXAMPLE
20
The Brandon Centre provides
counselling and psychotherapy to
young people between the ages of 12
and 21
Before
measure
After
measure
MST
Before
measure
After
measure
YOT usual service

21. MEASURING YOUR
IMPACT

22. QUALITATIVE OR QUANTITATIVE DATA OR
BOTH
To be informed by…
• the outcomes you want to measure
AND
• How accessible are your users/beneficiaries/stakeholders?
– How easy is it to get a representative sample? (for quant data)
– How easy is it for respondents to take part in research?
• How much time and resources do you have?
22

23. NO MAGIC RULE FOR SELECTING
APPROPRIATE TOOLS AND MEASURES
It depends on:
– ‘Fit’ between outcome and measure – will the data be meaningful?
– What is widely accepted?
– Cost / effort to collect, use, analyse data
– Cost/ effort from others to provide data
– Intensity and duration of programme
– Your appetite for rigour
• NPC preference for standardised measures that have been tested to robust
standards
23

24. TYPES OF QUANTITATIVE TOOLS & THEIR
USES
24
Type of tool
Measures
progress for
individuals
Aggregate
to show
change
Robust
measure of
change
Explain why
change has
happened
Admin. data ? ? ? 
Case work tool (e.g.
Outcome Star)
  ? 
Clinical tools    
Scales (developed
by researchers)
?   
Single questions ?  ? 
Hard outcomes
(e.g. statutory data)    

25. TYPES OF QUALITATIVE TOOLS AND THEIR
USES
25
Type of tool
Measures
progress for
individuals
Aggregate to
show change
Robust
measure of
change
Helps explain
why change has
happened
Interviews  ? ? 
Observation  ? ? 
Focus groups    
Ad hoc
feedback
?   ?

26. CASE STUDY
Scenario
• Intensive drama
programme with
young people.
• Programme lasts a
year and frequency
and duration of
contact is high.
• New and untested
approach but
charity believes it
to be very effective.
• Want to generate
evidence to
illustrate this
effectiveness and
expand the
programme to
other locations.
Young
people take
part in drama
workshops
focusing on
role play
Young people have
improved
communication skills
Young people have
more confidence to
seek help
Young people
have improved
behaviour
Young people learn
about different’
viewpoints
Improved
communication with
caregivers and
teachers
Young people get
the help they need
Young people learn
a new skill
Young people feel
better about
themselves
Programme using drama and theatre to improve communication skills and
behaviour of at risk youth

27. 27
The charity prioritised improved communication skills, confidence and
behaviour as outcomes to measure
Recommendation:
• Use short standard scales to measure these outcomes, pre and
post participation in the programme. Follow up one year after they
finish the programme.
• Use academic research to evidence the links between
improvements in these outcomes and reduced risk of anti-social
behaviour.
• Use these pre and post questionnaires with a random sample of
50 young people as it has limited resources to measure everyone
it works with.
CASE STUDY

28. ARTICULATING YOUR
IMPACT

29. 71
COMMUNICATING YOUR RESULTS
Knowing your audiences
• Commissioners
• Trusts & foundations
• Major donors
• Beneficiaries
• Volunteers
• Media etc.
• Regulators
• Trustees
• Management team
• Staff
Funders
StakeholdersOrganisation

30. WHAT DO YOU NEED TO COMMUNICATE
Five key questions:
1. What’s the problem you’re trying to tackle?
2. What are you doing to address it? (your activities)
3. What does that achieve? (your outcomes)
4. How do you know? (your evidence)
5. How are you learning & improving?
30

31. 3131
RESOURCES
• Website dedicated to Theory of Change: http://www.theoryofchange.org/
• NPC’s report on Theory of Change: http://www.thinknpc.org/publications/theory-of-
change/
• A good website to look for measures and indicators:
http://wilderdom.com/tools/ToolsSummaries.html
• http://www.performwell.org/ A website where you can find outcomes and standardised
scales relevant to a range of human welfare services
• The innovation network’s logic model workbook:
http://www.innonet.org/client_docs/File/logic_model_workbook.pdf
• The Centre for What works (http://www.whatworks.org/) has an outcome portal where
you can browse different outcomes and their corresponding indicators.
• TRASI (http://trasi.foundationcenter.org/) has a database where you can search
different approaches to impact assessment.
• The resource page of the Inspiring Impact programme (coordinated by NPC in
partnership with 7 other organisations), which includes a list of measurement tools and
systems http://inspiringimpact.org/resources/

32. RESOURCES
• Survey question bank: A website where you can research widely used surveys and
single questions by theme or using key word searches. http://surveynet.ac.uk/sqb/
• A list of the most common psychological scales and questionnaires:
http://www.ull.ac.uk/subjects/psychology/psycscales.shtml
• Ritchie et al. (2003) Qualitative Research Practice – A Guide for Social Science
Students and Researchers. Sage Publications Ltd
• How to communicate your results: Hedley, S et al (2010) Talking about results. New
Philanthropy Capital. http://www.thinknpc.org/publications/talking-about-results/
• A Journey to Greater Impact: NPC report profiling six charities who radically improved
their approach to impact measurement: http://www.thinknpc.org/publications/a-journey-
to-greater-impact/
• NPC’s mapping of outcomes for social investment: a number of report mapping
outcomes in different charity sectors and outlining appropriate measurement tools:
http://www.thinknpc.org/publications/mapping-outcomes-for-social-investment/
• Bradburn et al. (2004) Asking questions: the definitive guide to questionnaire design.
Jossey-Bass: San-Francisco.
32 3232