How to track, evaluate and evidence impacts arising from research. Presentation by Prof Mark Reed from Fast Track Impact

1. EVALUATING IMPACT

2.  Move to your break-out room
 Introduce yourself to your partner in 1 minute
 Swap places and let your partner introduce
themselves in 1 minute
 60 second countdown to finish up and make
any notes
 Come back to the main room and introduce
your partner in the chat
Evaluating Impact
Introductions

3.  In the chat now…
Evaluating Impact
Handouts and books

4. Evaluating impact

5.  The process of assessing the significance and
reach of both positive and negative effects of
research
 Your task is to identify causal links between:
 Research (cause)
 Impact (effect)
 To create an evidence-based argument that
your research was sufficient or necessary to
generate the claimed impact
Evaluating Impact
What is impact evaluation?

7.  Entry level evaluation: use common sense to
assess milestones and indicators (establishing
baselines as necessary)
Evaluating Impact
Evaluating impact

8. How
to
evaluate?
Reed et al. (2021) Evidencing
research impact : a
methodological framework.
Research Policy

10. Systems
analysis
Contribution analysis,
knowledge mapping,
Social Network Analysis,
Bayesian networks, agent-
based models, Dynamic
System Models, influence
diagrams, Participatory
Systems Mapping,
Bayesian Updating
Can be used in formative
or summative mode,
usually ex-post or during a
pathway to impact
Additive causation based
on tracing links between
causes and effects along
causal chains or pathways
to impact
• A significant
contribution made by
research to the solution
of a previously
intractable problem
• Increase and
strengthening of the
number of nodes or
connections in a social
network following a
participatory process
• Understanding of how a
group of actors relate to
each other and act
• Policy
• Other forms of
decision-making and
behavior change
• Capacity building
Textual, oral
and arts-based
Testimonials, ethnography,
participant observation,
qualitative comparative
analysis, linkage and
exchange model,
interviews and focus
groups, opinion polls and
surveys, other textual
analysis e.g. of focus
group and interview data,
participatory monitoring
and evaluation,
empowerment evaluation,
action research and
associated methods,
aesthetics, oral history,
story-telling, digital cultural
mapping, (social) media
analysis, poetry and fiction,
music and dance, theatre
Used either in formative
mode to enable
beneficiaries to engage
and shape feedback that
then enhances impact, or
in summative mode, ex-
post, to assess the extent
to which research
contributed to impact.
Causation is inferred by
building a case (sometimes
generative and sometimes
jointly with beneficiaries)
that triangulates multiple
sources of evidence to
create an evidence-based,
credible argument for
research being a sufficient
cause of impact.
• Testimonials or
statements from end
users (e.g. policy
makers) now applying a
modelling tool
• Testimonials from
practitioners explaining
how they gained a
higher level of
capability and capacity
handling daily work
thanks to a new
guidance (improved
skills, understanding,
and confidence levels)
• Improvements in
variables that indicate
the achievement of
goals set by a
stakeholder or other
social group who co-
produced research
(e.g. number of
community members
having acquired a
particular skill)
• Changes of perception,
awareness or attitudes
of a social group as a
result of engaging with
research
• Changes in culture,
cultural discourse or
appreciation and
• All types

11.  Evaluation design = research design
 Get win-wins for your research by asking “what’s
my impact” as a research question and identifying
methods already in your toolkit
 Get targeted help when there’s a tool missing
 Be proportionate
 Do parts of your design e.g. online survey,
interview
 Rigour from triangulation
 Get feedback, plugging gaps till it is believable
 Wait till you can do full design for high risk impacts
Evaluating Impact
How to evaluate

12. Evaluating Impact
Example 1
GDPR compliant permission to
contact sample of beneficiaries
Online survey, asking if they can be
contacted for interview
Interviews with sub-sample
1 2
3

13. Evaluating Impact
Example 2
Qualitative and quantitative
social media analysis
A funnel approach in
which you direct a
proportion of an
audience to a website
or other mechanism
where you can follow
up with them
Before/after polling of
media audiences
www.fasttrackimpact.com/
media-impact-guide-and-toolkit

14. Evaluating Impact
Example 3
Credible link between policy and
research (even if not cited)
Evidence
used in
credible
pathways
Testimonial to
prove causal
link
1 2
3

15.  Altmetric.com is useful for identifying social
media and other coverage of your research,
which you can follow up to determine if there
are significant impacts
 Policy documents citing your research
 For example…

18.  Remember: All forms of altmetrics tell you
someone is interested in your research, but
your task then is to find out why, and if they
are benefiting to assess impact

19. Evidence-based testimonials
Before you start: Ethics
approval?
Format:
• Part catch up and offer of
further help
• Part evaluation
1. Significance
2. Reach
3. Attribution
4. What could we have
done better?
Transcribe interview and
draft testimonial

20. Evidence-based testimonials

21. •“A research method in which the arts play a
primary role in any or all steps of the research
where art forms are essential to the research
process itself” (Coemans et al., 2015)
• Visual methods like photography, painting, collages,
murals
• Narrative methods like
poetry, fiction, comic strip
• Performative methods
like film, theatre, dance,
songs, music
Evaluating Impact
Arts-based evaluation methods
Dr Rachel Blanche
Queen Margaret University Edinburgh
Blanche, R. (2022) 'Arts based methods as a tool for impact research' [Video]. FastTrack Impact: Monitoring and
Evaluating Impact.Online. 28 February. https://eresearch.qmu.ac.uk/handle/20.500.12289/11963

22. • Benefits of arts-based methods (Reed et al., 2021):
• Create new knowledge spaces, elicit new perspectives
• Alternative (even non-verbal) means of expressing what
is difficult to conceptualise or articulate
• Effective way to explore, reflect, discuss, and describe
the aesthetic part of experiences
• Empower research participants
• Generate enriched data with additional layers of
understanding
• Create compelling evidence with the power to connect
with research audiences
Evaluating Impact
Arts-based evaluation methods

23. Formative feedback from a dance workshop
evaluating professional practice

24. “We invite people to annotate the
maps with their own observations,
landmarks, little known facts,
personal experiences, special places,
attractions and unattractions. In this
way we built up totally subjective
multi-authored psychogeographic
maps which become artworks in
their own right” (Williams & Teasdale
2013)
A key element of the project was The
Huntly Map, a large outline map of
the town (5ft x 7ft) containing road
names and landmarks such as parks
and rivers but no actual detail
• The map as an artefact
holds the evidence.
• It communicates that
evidence directly to the
viewer.
A psychogeographic* evaluation of impacts on community identity created an
experiential map of Huntly
* psychogeography: an exploration of a place that emphasizes interpersonal
connections to locations and routes

25. Who has a stake in my research?
Comments or questions?
Comment
in chat
Open
mic

26. Impact tracking

27. Read and discuss
Monitoring impact
 Find a way to continually track your
impacts easily to take the pain out of
reporting:
 Email impacts/evidence to yourself and file
 Shared Google doc or sheet
 Evernote: enable team members from any
institution to collate impacts in a shared
notebook without having to log into anything…

29. www.fasttrackimpact.com/evernote

30. Using evidence in REF case studies
Based on PhD research by Bella Reichard @BellaReichard based
on quantitative analysis of 217 and qualitative analysis of 180 of
the highest and lowest scoring cases, spread across Panels A, B,
C and D from REF2014

32. Quantitative linguistic analysis
1. Highly-rated case studies provided specific,
high-magnitude and well-evidenced
articulations of significance and reach
• 84% of high-scoring cases articulated significant and far-reaching
benefits, compared to 32% of low-scoring cases, which typically focused
on pathway
Phrases more common in high-
scoring:
• Significance and reach (specific and
high): in England and, in the US, the
UK’s, millions of, long-term, the
government’s, the department of,
the House of Commons, for the first
time, prime minister, select
committee
Phrases more common in low-scoring:
• Significance and reach (non-specific or low): a number
of, a range, nationally and internationally, in local, of
local, the north, city council, policy and practice, an
impact on, impact on the, the impact
• Pathways to impact: has been disseminated, disseminated
through, dissemination of, been disseminated, and
workshops, the event, the book
• Beneficiaries (not benefits): and community, practitioners
and, group of, members of the
Qualitative analysis

33. • 97% of high-scoring cases clearly linked the
underpinning research to claimed impacts,
compared to 50% of low-scoring case studies
• 42% high-scoring policy cases described policy and
implementation, compared to 17% in low
Phrases more common in high-scoring
• Attribution between research and
impact: cited in the, (was) used to
inform, to improve the, led to the,
resulting in, showing that, was
subsequently, produced by, reported in,
evidence for, cited in, led by
2. Highly-rated case studies established
links between research (cause) and
impact (effect) convincingly
Phrases more common in low-scoring
• Research outputs/process: the paper, peer-
reviewed, journal of, et al, research project, this
research has, by Dr, of Dr, research team
• Attribution between research and pathways:
work has, has informed, through the
0
10
20
30
40
50
60
70
80
Generally high
quality
corroborating
evidence
Some
questionable
quality evidence
Vague and/or not
clearly linked to
impacts
Number
of
case
studies
High-scoring Low-scoring
Quantitative linguistic analysis
Qualitative analysis
• Coh-metrix analysis shows higher-scoring cases had more explicit causal connections between
ideas and more logical connective words (and, or, but) than low-scoring cases

34. 3. Highly-rated case studies were
easy to understand and well
written
• Low-scoring cases more likely to have
academic phrasing: in relation to, in
terms of, the way(s) in which
• Flesch Reading Ease score, out of 100,
was 30.0 on average for 4* and 27.5 on
average for 1*/2* (all “college-graduate”
difficulty). Panels C & D high-scoring
case studies significantly easier to read
than low-scorers
• High scoring cases had more sub-
headings (especially pronounced when
comparing high to low cases in Panel D)
• High-scoring cases used more direct, plain language, had fewer expressions
of uncertainty or hedging statements, and were less likely to contain
unsubstantiated or vague use of adjectives to describe impacts
Qualitative analysis
Quantitative linguistic analysis

35. Anatomy of a claim
 Narrative structure
 Causal structure
 Claim structure: summary claim > specific claims
(significance + reach) > evidence for each

36. Who has a stake in my research?
Comments or questions?
Comment
in chat
Open
mic

37. Read and discuss
Choose between the following options:
 Main room: how would you prove that?
 Discussion about difficult to evaluate impacts
based on examples from the group
 Break out rooms: Design an impact evaluation
 Identify impact goal(s)
 Identify impact indicators per goal
 Discuss potential evaluation methods
 Choose a relevant mix of methods
Exercise

38. Read and discuss
A local example…

39. Next steps

40. Write in chat:
• What will I do, based on what I learned today?
Provide your email address and I’ll contact you a
month from now to remind you what you wrote
and see if I can help.
Or arrange to meet a colleague a month from
today to swap notes
Evaluating Impact
Actions
Comment
in chat

41. Get a reply from Mark to any query within 1 week:
send via Madie (pa@fasttrackimpact.com)
www.fasttrackimpact.com
@fasttrackimpact

42. Evaluating Impact
Feedback
www.fasttrackimpact.com/feedback-form