Presented to the NSW Marketing Working Group Seminar on Impact Measurement, November, 2017

1. Why Measure Social Impact?
7 November, 2017
Presentation for The State Library of NSW
Lena Etuk, CSI Senior Research Officer

2. 2
AGENDA
Introduction to the Centre for Social Impact
Key Terms
Why measure social impact?
What to consider for your journey into outcomes and
impact measurement

3. 3
THE CENTRE FOR SOCIAL IMPACT
Our purpose is to catalyse positive social change.
We do this by engaging and working with people, communities, and
organisations to grow their capabilities through research, education, and
leadership development.

4. 4
AGENDA
Introduction to the Centre for Social Impact
Key Terms
Why measure social impact?
What to consider for your journey into outcomes and
impact measurement

5. 5
SOME FREQUENTLY USED WORDS AND PHRASES
• Logic Models
• Inputs
• Outputs
• Outcomes
• Impact
• Impact assessment
• Outcome measurement
• Evaluation
• Monitoring

6. 6
LOGIC MODEL
Inputs
Provided
resources for
an
intervention;
e.g. budget for
education
Activities
E.g.,
construction
of schools
or hiring
additional
teachers
Outputs
Technical
results of an
intervention;
e.g. number
of new
schools
Outcomes
Direct effects
of an
intervention;
increase of
school
enrolment
Impact
Wider effects;
e.g.
contribution
to poverty
reduction
because of
improved
educational
outcomes
Your Planned Work Your Intended Results
If…then… If…then… If…then… If…then…

7. 7
OUTPUTS
The direct products of program activities and may include types, levels and targets of services to be
delivered. (Kellogg, 2004)
• Size and/or scope of the services and products delivered or produced
• Number of services or products delivered
• Participation rates
• Hours of service provided
For example:
• Increased community access and participation in health promotion and primary care services
• Evaluation and measurement program in place
• # of workshops attended

8. 8
OUTCOMES
Can be both the expected or unexpected effects of a program, initiative or activity.
Outcomes
• Knowledge
• Attitudes
• Values
• Behaviours
• Conditions
Different
stakeholders
• Clients
• Families
• Communities
• Funders
• Government
• Society
• Environment
Positive
Negative
Can be…
Short-term Medium-term Long-term
Occur at different
times

9. 9
Activity: Which of these are outputs and which are outcomes?
Outputs Outcomes
Number of audits completed
25 people attend awareness workshop
Visits to community centre / service provider
100 people access information line
Reduced social isolation
Improvement in family life & well-being
Increased level of confidence
20% improvement on knowledge scale
OUTPUTS VS. OUTCOMES

10. 10
SOCIAL IMPACT
Not clearly or consistently defined:
• Value created (Mulgan,2010)
• The broad or long-term effects of a project or organisation’s outputs, outcomes, and
activities. (New Philanthropy Capital, 2011)
• The portion of an outcome change that can be attributed uniquely to a program, that is,
with the influence of other sources controlled or removed. (Rossi, Lipsey and Freeman,
2004: 431)
• Positive and negative, primary and secondary long-term effects produced by a
development intervention, directly or indirectly, intended or unintended. (OECD, 2010:25)

11. 11
• Involves identifying how desired outcomes should be measured
• And involves collecting outcomes data
• Largely quantitative in nature
• Measures the prevalence of goal attributes among beneficiaries
Domains
Outcomes
Indicators
Measures
“75% of our program
participants are currently
satisfied with their health”
“On average, our program
participants had a moderate
level of knowledge about
financial products, now they
have a high level”
OUTCOMES MEASUREMENT

12. 12
“The systematic and objective assessment of an on-going or completed project, programme
or policy, its design, implementation and results.”
(OECD Development Assistance Committee, 2010:21)
• Process Evaluation // Outcome Evaluation // Impact Assessment
• Formative Evaluation // Summative Evaluation
Evaluation is a process rather than a specific method or approach.
Monitoring vs Evaluation
• Monitoring is on-going collection of data
• Evaluation is analysis of data to inform judgement
EVALUATION

13. 13
QUESTIONS ABOUT TERMS?

14. 14
AGENDA
Introduction to the Centre for Social Impact
Key Terms
Why measure social impact?
What to consider for your journey into outcomes and
impact measurement

15. 15
WHY MEASURE OUTCOMES & IMPACT?
We need to measure to understand if we are making
a difference.
• Are people really any better off?
• Are people more resilient, included and
connected?
• Have our actions had any impact?
Counting numbers of people and amount of activity
alone aren’t perfect indicators of results.
Why else might we measure
outcomes and impact?

16. 16
WHY MEASURE OUTCOMES & IMPACT?
Other benefits:
• Learning, development
• Communicating and branding
• Accountability and compliance
• Increased efficiency
• Organisational sustainability
Before we invest
millions, maybe we
should test it and see if
it works

17. 17
THE OTHER FACTORS THAT MAY MOTIVATE US
• Performance management & improvement
• Competitive market: informed consumers, facilitated choice
• Enable meaningful comparisons between programs & orgs

18. 18
WHY IS IT IMPORTANT?
A consistent approach to measuring outcomes and impact will help to:
• Clarify outcomes and results
• Inform decision making
• Help to work with others to address ‘wicked’ problems

19. 19
QUESTIONS ABOUT WHY WE SHOULD MEASURE IMPACT?

20. 20
AGENDA
Introduction to the Centre for Social Impact
Key Terms
Why measure social impact?
What to consider for your journey into outcomes and
impact measurement

21. 21
THE OUTCOME & IMPACT MEASUREMENT CYCLE
1. Understanding
2. Program
Model
3. Outcome
Measurement
Framework
4. Data
collection plan
5. Analysis and
communication
plan
6. Implementation
Plan
Implementation
w/continual
monitoring &
review

22. 22
UNDERSTANDING PHASE
• Clarify the measurement needs of the organisation or
program
– Information, evidence, questions
– Reflect on the role of context
• Consider existing external and internal systems,
frameworks, and reporting requirements
• Consider who needs to be involved in measurement, and
when
1. Understanding
2. Program Model
3. Outcome
Measurement
Framework
4. Data collection
plan
5. Analysis and
communication plan
6. Implementation
Plan
Implementation
w/continual
monitoring & review

23. 23
PROGRAM MODEL PHASE
• Clarify the rationale for the program, and
assumptions
– Theory of Change
• Articulate your 3 Ps:
– Purpose
– Process
– Progress
• Create a Logic Model (inputs, activities,
outputs, outcomes, impact)
1. Understanding
2. Program
Model
3. Outcome
Measurement
Framework
4. Data collection
plan
5. Analysis and
communication plan
6. Implementation
Plan
Implementation
w/continual
monitoring & review
Progress:
to what extent have
we achieved our
purpose/made a
difference?
Process:
how are we
going to
achieve it?
Purpose:
what are we
trying to
achieve?

24. 24
OUTCOME FRAMEWORK PHASE
• Clarify what the outcomes and impacts in your Logic
Model really mean
• Identify and select good indicators
– Measurable markers that demonstrate an outcome
• Identify and select valid measures
– The direct instruments that ascertain the existence of
something
1. Understanding
2. Program Model
3. Outcome
Measurement
Framework
4. Data collection
plan
5. Analysis and
communication plan
6. Implementation
Plan
Implementation
w/continual
monitoring & review

25. 25
DATA COLLECTION PLAN PHASE
• Only plan to collect what you will use
• Choose your data collection methods
– Outcomes can be measured in a variety of ways,
particularly when measured in the context of outcomes
evaluation
– Most common:
• Questionnaires or surveys designed in house
• Interviews
• Case studies
1. Understanding
2. Program Model
3. Outcome
Measurement
Framework
4. Data
collection
plan
5. Analysis and
communication plan
6. Implementation
Plan
Implementation
w/continual
monitoring & review

26. 26
ANALYSIS & COMMUNICATION PLAN PHASE
• Establish how to approach the analysis of data
• Identify appropriate channels for communication and
engagement
1. Understanding
2. Program Model
3. Outcome
Measurement
Framework
4. Data
collection plan
5. Analysis and
communication
plan
6. Implementation
Plan
Implementation
w/continual
monitoring & review

27. 27
IMPLEMENTATION PLAN PHASE
• A plan for how your organisation will roll-out outcome and
impact measurement
– Pilot vs. across whole organisation
– Capacity-building among existing staff vs. hiring new
staff w/expertise vs. procuring consultants
– Any work needed to improve the measurement or
learning culture of the organisation
1. Understanding
2. Program Model
3. Outcome
Measurement
Framework
4. Data collection
plan
5. Analysis and
communication plan
6.
Implementation
Plan
Implementation
w/continual
monitoring & review

28. 28
IMPLEMENTATION PHASE
• Measure the outcomes and impacts
• Communicate findings
• Adjust measurement activities, if needed
1. Understanding
2. Program Model
3. Outcome
Measurement
Framework
4. Data collection
plan
5. Analysis and
communication
plan
6. Implementation
Plan
Implementation

29. 29
QUESTIONS ABOUT THE IMPACT MEASUREMENT JOURNEY?

30. 30
REFLECTION QUESTIONS
• Are the expectations for outcome measurement understood? Internal and external
• Do your organisations support failure and learning? Are supports and expectations in
place?
• What would it take to develop a common language and understanding of the key
terms for outcome measurement across your organisations?
• Where do you think your organisations are in regards to readiness for outcomes
measurement?

31. 31
AMPLIFY SOCIAL IMPACT, A FUTURE RESOURCE FROM CSI
Tech platform to answer critical questions:
1. Where are social problems acute and
amongst which populations?
2. How do we know if we’re impacting the
problem?
3. Are our strategies and interventions
working and worthy of replication?
=
=
=
+ Social Progress Index

32. 32
CONTACT DETAILS
Lena Etuk
Senior Research Officer
P +61 2 8936 0929
E l.etuk@unsw.edu.au
@CSIsocialimpact www.csi.edu.au