1. SEDA Embedding Learning
Technologies
Evaluating & Sustainability
Brian Kilpatrick

2. Evaluating
• Motivation – why are we evaluating the project?
– Did you meet intended project objective
– Did you meet all/some of the project’s objectives/goals
outcomes? What didn’t you manage to achieve (this feeds
forwards into sustainability/closing/lessons learned i.e.
what is there still to do)
– How well did we meet those objectives i.e. to what
degree: just about; mostly; completely
– SMART targets (Specific Measurable
Achievable/Assignable/Attainable Realistic/Relevant Time-
bound)
– http://www.jiscinfonet.ac.uk/tools/smart-targets/

3. SMART targets
• Specific: targets a specific area for
improvement
• Measurable: quantify, or at least give some
indicator of progression
• A is for…achievable/attainable/assignable
• Realistic: state what results can realistically be
achieved, given available resources
• Time-bound: when will results/findings be
achieved

4. Leading and Lagging Measures
• Leading and Lagging Measures - what needs
to be measured to :
– Ensure objectives are being aimed for -
Performance Drivers (aka Leading Measures or
Indicators that can be used to plan and adjust
performance)
– Determine if the objective has been achieved -
Output Measures (aka Lagging Measures -
measuring events that have already happened)

5. Leading and Lagging Measures
• Objective - Getting a healthy suntan
• Measures to help ensure that we achieve
objective:
– Time in sun
– Amount of sunscreen used Leading
– Sunscreen factor Measures
– Application rate?
• Measures to determine if objective
achieved:
– Colour of skin
– Soreness of skin
– Distribution of suntan
Lagging
Measures

6. Evaluation - methods
• Focus Groups
• Surveys
• Observations
• Interviews
• Stat and system reports
• Feedback or Evaluation forms
• Benchmarking (internally across depts/externally
across institutions)
• Cost-Benefit Analysis ROI

7. Activity
• Split into groups of 3 to 4 people
• Select 1 project from within your group
• Using the Evaluation Matrix, determine which methods
you might use to evaluate or measure the success of your
project
• Draw a map of evaluation methods against your
measureable outcomes
• Remember of think about the following variables:
• Timing: when might you do that evaluation?
• Resources: what resource or tools will you need to do
the evaluation
• Frequency: will you need to carry out evaluation a
number of times and if so how often e.g. before and
after; weekly; after a specific event only?
Time:
• If you complete 1 move onto one of your colleagues’
15 mins
projects

8. Using your findings
How will your findings be used:
• Strategic planning e.g. devising policies, to feed into annual report
• Budgeting e.g. to inform budget requests, reallocate resources to better
meet user needs and expectations, or justify financial investment
• To secure funding from university administrators or external funders
• To identify problem areas and formulate plans to improve quality e.g.
improving staff customer service skills, improving user interfaces
• To identify staff training needs
• To inform and make changes to existing services
• For benchmarking purposes (which can help to share management
information, ideas and views, establish best practice, identify trends)
• To devise and effectively target publicity
• To identify areas for further research

9. Evaluation Cycle
PLAN
Decide on an evaluation method(s)
which suits needs and contexts e.g.
your stakeholders interests; project
goals.
ACT
do something with what you DO
found. Use it in annual review collect evaluation data (don't
and monitoring, but also use it forget other sources of data,
to make change/share good e.g. statistics or system reports)
practice.
STUDY
sort and reflect on what comes
out of the evaluation - this can
be done during the project or
at the end (continuous or at specific
points)

10. Sustainability
• Planning in conditions or processes that would
enable future success or continuation of the
activity
• Questions to ask yourselves:
– If I were not at Roehampton anymore would the
activity occur? Why?
– If not, then what needs to be in place for this activity
to occur without your specific intervention?
– Is that activity a core activity that I or my department
should be supporting/doing? If not then who?

11. Sustainability – how?
• Final report with recommendations
• Supporting resources/guidelines – what do
people need to know?
• Plans for resource discovery – how will people
find out about this?
• Dissemination – evidencing change
– Who needs to know
– Which committees/departmental/programme
groups should this go to

12. Sustainability – lessons learned
Documenting Lessons Learned
• Schedule time to keep track or address ‘lessons learned”
• Document lessons learned – keep a daily/weekly log if you
think that’s necessary
• Ask others to contribute i.e. stakeholders; participants
– Ask positive questions first: what went right?
– Progress to lessons from negative issues but keep positive e.g.
what you do differently/what was the most difficult challenge
– Ask stakeholders to talk about themselves e.g. what was their
role or perspective rather than being critical of others (could
they have done anything to prevent an issue)
• Share lessons learned – e.g. they can be fed into a future
knowledge base

13. Sustainability – lessons learned
What goes into a Lessons Learned log or report?
• Details of the event/lesson
• Stage when it occurred: Initiation; Planning;
Implementing/Delivery; Evaluation; Closing
• Effect or impact: e.g. was it positive or negative; severity of
impact
• Causes/triggers of the event: important to know why
something has or hasn’t occurred
• Indicators: were there any early warning signs that people
should be aware of
• Was it previously identified as a risk/is it new?
• Recommendations (for avoidance of issue in the future)

14. Sustainability – closing out
What do you need to do at a project end?
• Project Deliverables – obtain acceptance: where you have specific deliverables you
probably have ideas of what the success criteria for deliverables are and if they
have been achieved: do you need to receive acceptance from stakeholders? If so
what are the mutually agreed success criteria
• Get stakeholders to agree a signoff
• Document lessons learned – identifying what worked well, what didn’t, what could
be done better
• Final report – key purpose is to enable either the project to be run again or for
scaling up and passing onto the relevant team
– Was the project a success (summarise the output of your success measures)
– Document the dates when items where completed/delivered
– Document any variances [in terms of dates] with reasons for variance from
intended dates
– Summarise any significant changes in terms of deliverables
– Include other information e.g. lessons learned and significant risks particularly
ones that were not anticipated

15. Activity
• Split into groups of 3 to 4 people
• You have each been given some scenarios
• Take each scenario and discuss how you
could address the issues described
• Think in terms of what might be done in
advance of these situations arising
• You have 10 mins and then we will
reconvene to discuss with the whole group
Time:
10 mins

16. Scenario 1: introducing a new
technology
• Your team has run a successful pilot of an online
grade management system: you’ve exceeded
your outcomes and met all stakeholders
requirements. However, when you’ve presented
findings to your head of department they
respond that there’s no budget to take this
further at this point in the year.
• Q. What would you do next and how might this
situation have been avoided?

17. Scenario 2: embedding an existing
technology
• You've run a series of awareness raising sessions about
using audience response systems in class for staff
across academic departments. The sessions went well
but on the feedback forms for the session, many staff
express a lack of confidence in the software working
well in teaching rooms/consuls as they've heard other
staff have had issues and are also wary of conducting a
session unaccompanied or without first hand support.
• Q. How might you address this issue? Discuss the steps
you could take and plans that could be put in place.

18. Scenario 3: training based project
• You successfully piloted a series of online student
inductions - feedback was positive and observations of
students doing certain set tasks affirms that their
performance improves after attending the induction. The
resources are in place and you are ready for the inductions
to 'go live'. However, when you go to promote and embed
these sessions you find resistance from programme teams
to accommodate any introduction during Induction Week
as they already have a packed schedule and they aren't
supportive of giving up any class time.
• Q. How would you approach successfully communicating to
students about the resource and promoting its benefits?

19. Summing up
• Review Project Outcomes
• Set SMART targets to be measured or evaluated
• Choose the appropriate method(s) for evaluating
• Record your lessons learned: evidence
• Anticipate issues that my affect sustainability and
plan for them
• Determine your future role and where service
ownership will lie
• Use the evidence gathered to generate
recommendations in your final reports

20. SEDA Embedding Learning
Technologies
Evaluating & Sustainability
Brian Kilpatrick