Workshop presented by Professor Jane Seale, hosted by LTRI and ALT. MArch 20th 2012

1. Making a difference: How can
we demonstrate the impact of
learning technology research?
Professor Jane Seale

2. Overview
• Provide an overview of the different ways that funders
such as ESRC and HEFCE define and conceptualise
“impact” and “users of research”;
• Offer my personal interpretations, with examples, of
how learning technology research might demonstrate
impact;
• Facilitate debate and reflection regarding how well
placed current UK learning technology research is to
meet the challenges of demonstrating impact.

3. Defining impact
• Broadly speaking:
– Where research has a transformative and beneficial
effect on people, groups, organisations, communities
– Evidence for this impact goes beyond citations in
academic journals
• Impact on “real-world” users- through user
engagement

4. Separating impact of e-learning from the
impact of e-learning research
E-learning has had an impact
• We “know” that e- on pedagogy- it is
learning can be motivational, teachers have
transformative been transformed
• How can we
demonstrate the impact
of research that reveals
this transformation?

5. What claims for impact can the e-
learning research community make?
• Conceptual
– Diana Laurillard (2002) Rethinking university teaching:
a conversational framework
• 4350 Google Scholar citations
• Practical
– Gilly Salmon (2003) E-moderating: The key to
teaching and learning online
• 2676 Google Scholar citations
• Can “we” prove that government, universities, lecturers
have transformed the way we teach using technologies
as a result of these publications (through changes in
policies, standards, practice, systems, resources etc.) ?
– How??

6. What claims for impact can I personally
make? A personal reflection (1)
• Jane K Seale, Alan J Cann (2000) Reflection on-line or off-line: the
role of learning technologies in encouraging students to reflect:
Computers & Education, 34, 3-4, 309-320
• Empirical paper: one of my papers submitted to RAE 2001 as part
of Kings College submission
– Comparison of two individual teaching practices
• Dissemination activity: 2 conferences, Alan's‟ personal web page,
2nd related journal paper
• Impact ??
– CAL99: positive reaction regarding “honesty” in admitting failure
– CTI Biology Virtual Conference 1997- voted best paper
– My most cited paper: 74 Google citations (all academic journals
and conference papers)

7. A personal reflection (2)
• Phipps, L., Sutherland, A and Seale, J (eds). Access All
Areas: disability, technology and learning. ALT/JISC/TechDis.
(2002)
– An edited pamphlet aimed at practitioners- review/description of
accessibility law and implications for practice
– Not submitted to RAE 2008 by Southampton University
• Dissemination (impact)activity: freely available online
through TechDis , ILT, ALT
• Impact??
– 6th most cited publication in Google Scholar (mixture of
academic and policy related citations)
• Equality Challenge Unit (2006) Disability legislation: practical guidelines for
academics..
• Welsh Assembly Government (2005) Disability Issues for Post-16 Learning
Provision.
• Cited in many new lecturer programme reading lists

8. Personal reflections (3)
• LEXDIS- JISC funded project 2008
• Students and academic staff as
partners in the project- user
engagement
• Dissemination (impact) activity:
– Project website, 3 journal articles
(empirical)
2 newsletter articles, 1 book
chapter, 5 conference papers,
workshops, desk-side 1-to1
• Impact??
– One whole page spread in TES
– Students still contributing to the
website 4 years after end of
project
– Winner of 2009 IMS Learning
Impact Award

9. Which of these three projects had the
most “impact”?
• I personally value each of these 3 pieces of
work for different reasons in relation to their
impact on academia, policy, & students
• Link between impact and quality potentially
complex
• Only the last project had anything like a
concrete strategy for user engagement at the
beginning of the project

10. 1. How do funders such as
ESRC and HEFCE define and
conceptualise “impact”?

11. Impact and Excellence
• Both HEFCE and ESRC assert that you
can‟t have impact without excellence
– Excellent research underpins excellent
impact

12. Impact is about showcasing and
exploitation
• HEFCE: The REF will
– Showcase the success of UK research in
contributing to the economy and society
– Encourage more effective dissemination,
application and exploitation of research

13. Impact is about justification of
value and worth:
• ESRC:
– “In recent years, the government has placed
increasing emphasis on the need for evidence of
economic and social returns from its investment in
research.
– By ensuring that ESRC-funded research makes the
biggest possible impact on policy and practice, and
improving how we measure and capture this, we are
better able to support the case for research funding.
– Impact helps to demonstrate that social science is
important – that it is worth investing in and worth
using.”

14. Looking forwards and looking
backwards
• ESRC : when bidding: accounting for
future impact
– ESRC/RCUK: applicants are asked to give an
account of the anticipated future effects of their
research on potential non-academic users
• HEFCE REF: accounting for past impact

15. HEFCE (REF) definitions
• For the purposes of the REF impact is defined as an effect on,
change or benefit to the economy, society, culture, public policy or
services, health, the environment or quality of life, beyond
academia
• Having an effect on:
– the activity, attitudes, awareness, behaviour, capacity,
opportunity performance, policy, practice, process or
understanding
– of an audience, beneficiary, community, constituency,
organisation or individuals
– in any geographic location, whether locally, regionally,
nationally or internationally
• Impact on academic knowledge excluded
• Impacts on students, teaching or other activities within submitting
HEI excluded

16. REF impact assessment: case studies
• Provision of “case studies” in which starting
point is 2* paper(s) and the HEI provides a case
for:
– Impact of the research described in the paper(s)
– What the HEI has done to facilitate impact of the
research
• 20% of overall assessment score
• 2 case studies per 15 FTE (6 for 45+ FTE)
• Jan 1st 1993- 31st December 2013

17. Impact Criteria (drawn from Panel C
interpretation)
• Reach: understood in terms of the extent and diversity
of the communities, environments, individuals,
organisations or any other beneficiaries that have
benefitted or have been affected
– Reach will not be assessed in purely geographic terms, nor in
terms of absolute numbers of beneficiaries, but rather in terms
of the extent to which the potential number or group of
beneficiaries have been affected
• Significance: The degree to which the impact has
enriched, influenced, informed or changed policies,
opportunities, perspectives or practices of communities,
individuals or organisations
• Panels B & C will assess the two criteria on a holistic
basis

18. Type Examples
Impacts on creativity, Informed public or political debate
culture and society Improved welfare, equality, social justice,
Production of cultural artefacts (films, TV)
Economic, commercial, Development of new or improved products or processes
organisational impacts Developing alternative economic models
Changed approach to management of resources resulting
in improved delivery
Impacts on environments Changed resource management practices
Health and welfare Influence on CPD/improved training standards
impacts (incl. protection Improved provision or access to services
or advocacy of rights or Improved health and welfare outcomes (must include
interests) educational outcomes!)
Impacts on practitioners Development of resources to enhance professional practice
and professional services Use of research findings to define best practice or in
conduct of practice
Influence on professional standards, guidance or training
Practitioner debate has been informed or stimulated by
research findings
Impacts on public policy, Influencing the work of NGO or commercial organisations
law and services Policy debate stimulated by research evidence
Improved public understanding of social issues/challenging
conventional wisdom

19. What counts as evidence?
• Qualitative or quantitative for Panel C
• For Panel B- wherever possible
quantitative should be included

20. Example indicators of impact
• Citations
– Citation in a public discussion, consultation document or judgement
– Citation by journalists, broadcasters or social media
– Citation by international bodies such as UNESCO, IMF
– Evidence of citation in policy, regulatory, strategy, practice documents
• Measures
– Number of visitors, audience, participants
– Measures of improved inclusion, welfare or equality
– Service satisfaction measures
– Outcome measures
– Quantitative data relating to cost-effectiveness or organisational
performance

21. Examples of impact indicators
• Debate
– Evidence of debate among practitioners leading to
developments in attitudes or behaviours
– Public debate in the media
– Parliamentary or other democratic debate
• Use
– Use in scrutiny or audit processes such as Select
Committees
– Incorporation in training or CPD material
• Reviews and documentation
– Media reviews
– Independent documentary evidence of influence on
guidelines, legislation, regulation, policy or standards
– Documented change to professional standards or
behaviour

22. Evidencing impact
• Each case study must explain how the
research led to or contributed to the
impact and include appropriate sources of
information external to the HEI to
corroborate these claims

23. RCUK/ESRC definitions of impact
• “Research Councils UK (RCUK) defines research impact as
'the demonstrable contribution that excellent research makes
to society and the economy'.
• Research impact embraces all the diverse ways that
research-related skills benefit individuals, organisations and
nations. These include:
– fostering global economic performance, and specifically the
economic competitiveness of the United Kingdom
– increasing the effectiveness of public services and policy
– enhancing quality of life, health and creative output.
• A key aspect of this definition of research impact is that
impact must be demonstrable. It is not enough just to focus
on activities and outputs that promote research impact, such
as staging a conference or publishing a report.
– You must be able to provide evidence of research impact, for
example, that it has been taken up and used by policy
makers, and practitioners, has led to improvements in services or
business.”

24. ESRC Types of research impact
• Academic impact is the demonstrable contribution that
excellent social and economic research makes to scientific
advances, across and within disciplines, including significant
advances in understanding, method, theory and application.
• Economic and societal impact is the demonstrable
contribution that excellent social and economic research
makes to society and the economy, of benefit to individuals,
organisations and nations.
• The impact of social science research can be
categorised as:
– Instrumental: influencing the development of policy, practice or
service provision, shaping legislation, altering behaviour
– Conceptual: contributing to the understanding of policy issues,
reframing debates
– Capacity building: through technical and personal skill
development.

25. Planning research impact
• “To plan impact effectively you need to:
– identify your key stakeholders, for example, other
researchers; public sector; business/industry
– Identify how they will benefit from your research – types
of impact might include: improving social welfare/public
services; influencing public policy; contributing to
operational/organisational change
– Identify how you will ensure they have the opportunity to
benefit, for example through organising public events;
conferences; interaction with the media.
– For practical guidance on planning research impact, see
the information on developing an impact strategy”
http://www.esrc.ac.uk/funding-and-guidance/tools-and-resources/impact-toolkit/

26. ESRC Impact strategy: objectives
• “What are the likely outcomes of this research?
• Who will benefit from this research?
• How will they benefit from this research?
• How can you involve potential beneficiaries in
this research?
• How will you know if it has made a difference?”

27. ESRC Impact strategy: Messages
• “Clear, succinct messages that summarise your research.
• Over-arching messages that can be used while it is underway, or
specific messages that relate to particular parts of the project.
• Using different formats
– a media release
– a report
– a research briefing
– a newspaper article
– a website page.
• Think in advance about stories, case studies and 'packages' of
information that will bring your project to life for key audiences.
• Creating a brand: distinct identity
– your project name
– a 'strapline' or one-line description of the project
– a logo or visual mark,
– the application of your brand across a range of materials
– Ensure your brand reflects the overall values and objectives of your
research and your impact plan.”

28. ESRC Impact strategy: targeting
audiences
• “It is vital to know who your key audiences
are.
– Think beyond the obvious organisations
and individuals.
• Prioritising your audiences
– Since time and money are limited, it is useful to
rank each of your potential audiences and user
groups according to their importance and
influence relative to your strategy.
– A good question to ask is: if you had half the
money and half the time to spend on your
strategy, which of your potential audiences
would you focus on?”

29. ESRC guidance on tools and channels
• Branding
• Events
• Interactive media: websites, social media
• Media relations
• Networking
• Public affairs: influencing policy makers
• Publications
• Public engagement: shape research agenda, steer project, share
results, raise awareness

30. Evaluating impact
Evaluating economic and capacity building impact probably quite
straight forward. For example, for capacity building could apply this
ESRC advice:
• “Quantitative data– provide measures, for example of how many people
attended the event and what they thought of it.
• Qualitative data – seek to illuminate individual experiences and provide
additional subjective context to the evaluation. These data explore the
participants‟ experience in more depth than quantitative data.
• Observational data – exploring how people participate in an event can be
illuminating. Did they participate in all events/activities? Were some
aspects more popular than others? How did people interact with your
website or display? A clear idea of what you are looking for is important
when structuring observational data.
• But, is this measuring user engagement and can this be
legitimately used as a proxy for impact”

31. Evaluating impact
• Evaluating impact of conceptual and related
impacts harder and ESRC is addressing this
• It is commissioning case studies on methods for
capturing the more diffuse types of social science
impact
– e.g. mapping conceptual developments (such as changes
in thinking, debate, culture and direction)
– „people-flow‟ impacts across the researcher/user interface
•

32. Implications for e-learning research
• If we want to defend or protect e-learning
research then we have to demonstrate its
public value

33. 2. How can learning
technology research
demonstrate impact?

34. TLRP TEL as an example
• TEL and impact
• “The TEL programme is keenly aware of the need to demonstrate
impact beyond academia. A number of projects have already
begun to demonstrate how their TEL funded research could have a
practical, positive impact on such areas as:
– relieving teacher‟s time and paperwork constraints
– addressing difficult subjects in schools in different ways
– providing better opportunities for disadvantaged people to
develop skills through interacting with technology
– showing how technology can be used in learning environments
to save time and money whilst providing improved learner
outcomes.

35. Innovations in translation
• http://tel.ioe.ac.uk/category/impact
– Exhibitions
– Prizes
– Practitioner workshops
– Media articles
– Journalist informed tweets
– Videos
– Press releases
– Breakfast conference with MP‟s

36. Haptel as a case in point
• Clear messages about:
• Cost effectiveness
• Evidence that students do
just as well with haptics
• User engagement- clinical
dentists on the team
• Vision for implications for
future

37. Haptel as a case in point
• Participated in an
evening of
demonstrations of the
latest medical
technology for the
general public at the
Science Museum

38. 3. Facilitate debate and
reflection regarding how well
placed current UK learning
technology research is to meet
the challenges of demonstrating
impact.

39. An impact SWOT analysis for e-
learning research
SWOT Examples
Strengths We have a detailed knowledge and insight into
one key user group- practitioners.
Weaknesses Poor relationship with government-policy
naivety
Tendency to follow technological trends rather
than set them
Opportunities Methodologies suited to user-engagement
Knowledge transfer
Threats The culture of the lone e-learning “champion”
researching own practice
The memory of the failed UKEU- the elephant
in the room

40. Strengths
• We have a detailed knowledge and insight into one key user group-
practitioners
– We inhabit their world because more often than not we are “one
of them”
– Ideally placed to evidence impact on practice and inform
practitioner debates, practice, standards…(but we have to think
beyond our own institutions to do this)
Impacts on Development of resources to enhance professional
practitioners and practice
professional services Use of research findings to define best practice or in
conduct of practice
Influence on professional standards, guidance or
training
Practitioner debate has been informed or stimulated by
research findings

41. Weakness
• Very little e-learning research(ers) speak to
government
– What e-learning report/papers have been cited in
key government education policy documents and
debate?
– Web science on the other had/have the ear of
government
– TEL/ALT working hard to cement a good relationship
with BIS on our behalf
• But we have to engage too…..
• To do this we have to re-conceptualise who we think our
end-users are

42. Weakness
• Tendency to follow technological trends rather
than set them:
– We apply the tools around us to educational contexts
(e.g. web 2.0 tools) whilst there is some innovation in
this (applying existing tools to new situations) but
couldn‟t we be doing so much more?
• Naïve and ill-prepared in our approach to
commercialisation of technological
developments

43. Opportunities: methodologies suited to
user engagement
• Channels of engagement- e.g. social media
• Process of engagement- participatory action
research, partnerships with industry

44. Methodologically, impact requires us to step
out of our comfort zones and make “contact
with strangers”
• Brewer (2010) Impact requires the discipline to reconsider its own
„strangers‟, those with whom research relationships came to
appear unusual and odd […] more obviously, policy-makers and
civil society groups [..]
• Impact has accelerated contact with „strangers‟ and is disquieting
for this reason, but the long-established tradition of action
research, participatory forms of research, or research sensitive to
respondents‟ sense of research fatigue, are forms that enhance
people‟s participation in the design and conduct of the research
precisely in order to increase its impact

45. Opportunities: Knowledge transfer
• Can our work translate to related fields?
• Examples:
– Work based training- internships (current political hot
potato with introduction of 9K fees)
– VLE design (why shouldn‟t we shape what
Blackboard does for once?)
– Distance learning design and delivery for military
personnel on duty

46. Threats: burned fingers and long
memories
• UKEU and long memories
– A company and website set up by HEFCE that
promoted online degrees from UK universities.
UKeU delivered courses over a learning environment
developed by Sun Microsystems UK.
– In 2004 it was announced that the project should be
wound up as it was a failure having cost £50 m and
recruited only 900 students
– E-learning a victim of its own “hype”

47. Threats: Lone Ranger syndrome
• Brewer (2010)
• Impact disturbed the comfortable rules of the
game by which research reputations and
resources were established
– Marks the demise of the lone researcher model of
social research in the 21st century.
– Promotes engagement with real world problems

48. Threats: Lone Ranger Syndrome
• We need more than ever to develop
research teams that possess the wide
range of skills required to conduct
research AND plan, implement and
evaluate an impact plan
– Researching own practice, in own institution
through personal interest and enthusiasm,
with limited skill-set no longer “cuts it”

49. Thank you, questions,
comments, reflections,
observations