Despite Learning measurement frameworks like Kirkpatrick and Philips ROI that have been around for so many years, defining meaningful measures still continues to be a big challenge for L&D. ‘E’ has become the most ignored in ADDIE.
We talk to an experienced panel about the unsatisfactory current state and what needs to be done to improve the importance, visibility and value of measuring Learning effectiveness in the real world. We focus on how to collect actionable insights about learning effectiveness: not only seeking to answer, did learning achieve the desired or intended result, but also why/why not.
2. Introductions
AbbVie
ABC
AGL
AMP
ANZ Bank
ARCA
Australian Institute of
management
Australian Red Cross
Bank Australia
Bankwest
Blackboard
BNZ
BT Financial Group
Cancer Council Queensland
CBA
CBH Group
Coca-Cola Amatil
Concordia University
CUB PTY LTD
Degreed
Downer
Energy Australia
Ericsson
Ernst & Young
FACS
FedEx Express
Ford-Werke GmbH
HCF
IAG
IMC
IP Australia
Justine Jardine
Lynda Moe Transformations
Manipal Global Ed Serv
McGrathNiccol
Melbourne Business School
NAB
NATA
Novartis
Presence of IT
Primary Health Care
Qantas
Qantas Credit Union
Qatar Airways
QBE
Queensland Public Services
RACP
Recoveriescorp
Rio Tinto
Shine Laywers
Siemens Ltd
Simplot
Skillsoft
SMS Management & Tech
St John Ambulance WA
Suncorp
Telstra
Toyota
TP3
University of Sydney
Westpac
WVI
Attendees From140+
Vanessa Blewitt
Learning Effectiveness Lead,
Corporate Training and
Learning – Nestle
Will Thalheimer
President – Work-Learning
Research, Inc
Jeevan Joshi
Producer – LearningCafe
5. Next Webinar - VR in Learning – Has its time come ?
http://bit.ly/lcwebmay16
• State of VR (Consumer & business)
• Use cases for business
• How to develop a VR Learning
solution
• Opportunities and challenges
Thurs May 12th, 2016 12 to 1 PM
Kain Tietzel
Founding CEO –Start
Plus others
Why you should attend
• 2016 is supposed to be year of VR
• L&D likely to asked about it.
• VR Learning is being built.
• Stay ahead of curve.
6. Melbourne UnConference – Aug 18, 2016
Limited to 100. Tickets available. Sydney was sold out
http://www.learningcafe.com.au/events
Tony Gleeson
CEO - AIRAH
Julie Catanach
Exec Manager
Performance
Suncorp Group
Peter Hall
Head L&D
QBE - APAC
David Le Page
Comm Mgr -
Melbourne
Andrew Wright
Worldwide
Intranet
Challenge
8. Monthly Observations in Australian L&D*
•Significant restructuring in HR/ L&D.
• Head count reductions
• Getting new skills in the team.
•Gap in Business Acumen in L&D, a real worry.
•Major LMS & HRMS not keeping pace with
flexibility and UX.
• Number of point cloud solutions coming into the market .
•More focus on hard skills / business Learning.
• Less on soft skills.
* Opinions based on discussions and interviews with senior L&D professionals. Not survey involved.
9. Webinar Dashboard
Start Up/
Experiment
Organisation / Industry
Adoption
Why Bother ? Remove Hype
Maturity
Business
Value
Low
Low
High
High
Now
Novice Explorer Practitioner Expert
Get
Answers
Make
Sense
Looking for Answers
Expertise Level
Next Steps ?
Not Much A Lot
Vision
LearningCafe Expertise
Run workshops for L&D teams for
• Leading Australian Bank
• Leading Insurance Company
Consulting on architecture for
• Professional services firm
Covered in UnConference 2015 + 2016
10. Questions from you
• What are 3 tips in gathering valuable data/statistics?
• How do you measure accurately if for e.g. only 100 out of
500 people respond to a survey? When should you make
assumptions from the data?
• Like quality, learning effectiveness is also a perception in
the minds of key stakeholders (and influenced by their
expectations). How do we best accommodate this reality
in our measurement strategy?“
• Value in blended learning design - at the end, throughout,
at the start or combination of?
13. PROBLEMS
Too difficult to
isolate impact of
Learning
Evaluation is not
standardized
enough
Not enough
budget and skills
Business moves
fast and loses
interest.
17. TinCan/xAPI@Work
www.tincanatwork.com
17
Intranet – Videos/podcasts
Discussion Boards (COP)(yammer)
Wiki (Aussie /prod info/Wiki/IT)
On job activities (Perf Management)
Discussions
Team meetings (learning component)
Other business systems
Online Courses on LMS + Webinar
Classroom Courses/ Online
Coaching
On Job Activities (competence demo)
Assessments/surveys (e.g. survey
monkey)
Simulation
Performance Support Tool
Secondment
Social Media
YouTube
Facebook
Website – HBR
Any web resource
External COP
Coffee conversations
MOOCs
External courses
Formal Education (Degrees)
Formal Collaborative Program with
Unis
Memberships/prof bodies
Formal mentoring programs
Certifications/exams
Formality
Sources
Internal
Informal
External
Formal
18. xAPI/ TinCan – Solution ?
Layer 1
• Modern version of
SCORM
Layer 2
• Record any
learning
experience
Layer 3
• Frees data from
LMS
Layer 4
• Correlate job
performance and
training data
http://tincanapi.com/the-layers-of-tin-can/
Happening Aspirational
Technical Strategic
Low readiness Maturity /capability
www.tincanatwork.com
21. Deciding to Measure Learning Effectiveness
What is the value of T&L investments?
Executive Board
• That was the easy part!
22. Starting Point:
T&L as a “HR Thing”, an Event
• < 1 in 10 learners
discuss expected
outcomes with line
manager.
• Alignment to
business not clear.
• Alignment to learner
development plan
low.
• Key measure = • No on the job follow-
up or support.
• Action plans un-
actioned.
• No measures of
what happened…or
why.
• “Black box”.
Before During After
Look Familiar?
23. Defining “Value”
Guiding Principle:
In any transaction, the supplier determines the
price, but the purchaser determines the value.
In order to measure value, first need to create conditions for value.
Tools, Process, Measures
Design Delivery
What is the
value of T&L
investments?
Collect
Nestle
Insights
Conduct
Industry
Research
Develop
Learning
Effectiveness
Framework
24. Defining Learning Effectiveness
Measures: Design
Design
With the business for the business
Business Level Measures
WHAT: What is expected?
What does success look like?
How will you recognize it?
Why this learning…target which business
objectives & priorities?
What will you reflect on in 6 months time
to decide if learning was worthwhile/
added value?
Expected outcomes? … observable /
measurable? KPI impacts? Trends?
Feedback? What should CHANGE?
WORKSHOP
Senior Stakeholders
& Alumni
H
O
W
&
W
H
O
INTAKE FORM
Senior Stakeholders
& Alumni
INTERVIEW
Reflections & Feedback
Alumni @+200 days
25. Defining Learning Effectiveness
Measures: Delivery
Delivery
For on the job performance change
Learner Level Measures
WHAT: What is expected? What happened?
WHY: Why did it happen?
Learning Alignment
• To development plan & objectives
Learning Impact on business
• Expected outcomes e.g. KPIs
Learner Development
• On the job behaviors – do & say
Learning Application
• Action Plan status
• Opportunities & Obstacles
Learner Experience
• Confidence
• Commitment
• Net Promoter Score
H
O
W
&
W
H
O
Surveys
Learner with Line Manager
Option: HRBP Facilitation
@ Before @After
Surveys
Learner
@Nomination @During
No survey is called “survey”
Learner centric questions
26. Importance of Why Measures
Example of Actionable Insights
WHAT
Happened?
• Proficiency Gap Closure = 50% v Expectation (Target) of 80%.
WHY
Did it happen?
• Action Plan Status = 60% Fully or Mostly Implemented.
• Obstacles to apply learning on the job = 70% learners reported
higher/competing priorities.
WHAT
Action to take?
• Current Cohort: Targeted business escalation, coaching.
• Future: Increase business to business communication re importance of
application to achieve business and career success, learn &
leverage from strength business areas, add how to address
higher/competing priorities to learning experience.
Target
Set benchmarks & change
variances
Track Trends
Over time.
Avoid “knee-jerk” reactions
Tell
Tailor to stakeholders
Apply “So What?!” Filter
27. Challenges: Organisation &
Process, Data, Enablers
Organisation
• Identifying right stakeholders & getting them to the table.
• Defining what success looks like in a way that is tangible &
linked to business.
• Defining objectives to support learning effectiveness
measures (finite, verb, scalable for proficiency levels).
Process, Data, Enablers
• Data access & quality for analytics.
• Sophistication of requirements v enabler capabilities.
• Managing surveys & reporting > timeliness, quality, defining
use cases.
• Scalability.
28. Challenges: Climate & Culture
Climate & Culture
• Survey response rates (L&D to business).
• Training as reward & recognition.
• Fear of “why” measures (e.g. manager support rate).
• Learners: Lack accountability for self-development.
“Develop me.” “I was told to come”.
• Line Managers: T&L = . Develop people not include
in bonus. “ I don’t have time”.
• HRBPs: Operational focus. “Not My Job”, “Too hard”,
“Good training gets applied”, “Why measure?”.
• L&D: Event focused. Self-contained. Service oriented.
Uncomfortable to challenge business. “Not what I’m
asked for”, “No influence beyond event”, “What am I
supposed to do with this?” “What if I’m told to “fix it?””
• ALL: OVERWORKED. OVERWHELMED. CHANGE
RESISTANT.
29. Top 10 Key Learnings
WHAT
“as expected?”
!As defined by
stakeholders!
WHY
“why / why
not?”
Act to achieve
/ exceed “as
expected”
!As defined by
stakeholders!
Effective learning
effectiveness
measures
1. “What does success look like?” is a hard question to answer!
ASK! Don’t ass-u-me.
2. “Fear” of what measures will reveal & accountability for action
3. Required culture change in L&D as much as business
4. Use inspiring facts & stories. Shine a light: “look familiar?”
5. Appetite exists & can be built. Don’t push. Leverage multiple
channels to “build the army” of collaborators & early adopters.
6. Engaging stakeholders increases stakeholder engagement &
accountability
7. Respectful challenge is well-received, build & helps earn a seat at
the table
8. Alignment & behaviour change more valued than ROI
9. Focus on HOW more than WHAT
10. Long process of CHANGE. Be the tortoise.
+1 Start small & focussed
31. Questions from you
• What are 3 tips in gathering valuable data/statistics?
• How do you measure accurately if for e.g. only 100 out of
500 people respond to a survey? When should you make
assumptions from the data?
• Like quality, learning effectiveness is also a perception in
the minds of key stakeholders (and influenced by their
expectations). How do we best accommodate this reality
in our measurement strategy?“
• Value in blended learning design - at the end, throughout,
at the start or combination of?
33. Next Steps
Join Special
Interest
Community
Attend Workshop
Content Curation
& Dissemination
for Learning
Attend
UnConference
Melbourne
Brisbane
LearningCafe
LinkedIn Subgroup
Register interest
www.learningcafe.com.au
Register interest
www.learningcafe.com.au
Or send us an email - enquiry@learningcafe.com.au
35. Next Steps
Join Special
Interest
Community
Attend Workshops
Attend
UnConference
Melbourne
Brisbane
LearningCafe
LinkedIn Subgroup
Register interest
www.learningcafe.com.au
Register interest
www.learningcafe.com.au
Or send us an email - enquiry@learningcafe.com.au
Notas do Editor
What are the current approaches/ trends for managing change for Learning initiatives ? We look at contemporary Change Management frameworks including the Burke and Lewin Model of Organisational Performance and Change.
Their experiences including:
Approach for designing and implementing a Technical Academy.
Change management for learning technology projects in financial services.
Change approach when Learning is integrated with other HR processes for large transformation initiatives.
Many Learning professionals are skilled in change management (and visa versa). Does this result in better results due to closer integration ?
Do the change approaches differ for technology enabled learning ?
Are the change approaches for Learning meeting the needs to be responsive and agility ?
Opportunities
Develop stronger business cases to secure, maintain resource allocation to L&D
To demonstrate L&D activity as synchronised to operations and overall organisation goals
Better secure significant investment in new HRM initiatives through CAPEX
Challenges
Deciding upon the most critical measures (“vital few versus trivial many”)
The financial literacy of L&D (and HR staff more broadly) is highly variable across organisations
Proving links between the training intervention/s and the operating and business outcomes
“how confident am I that the increase in sales was due to training as opposed to advertising/promotional activity”
Opportunities
Develop stronger business cases to secure, maintain resource allocation to L&D
To demonstrate L&D activity as synchronised to operations and overall organisation goals
Better secure significant investment in new HRM initiatives through CAPEX
Challenges
Deciding upon the most critical measures (“vital few versus trivial many”)
The financial literacy of L&D (and HR staff more broadly) is highly variable across organisations
Proving links between the training intervention/s and the operating and business outcomes
“how confident am I that the increase in sales was due to training as opposed to advertising/promotional activity”
Opportunities
Develop stronger business cases to secure, maintain resource allocation to L&D
To demonstrate L&D activity as synchronised to operations and overall organisation goals
Better secure significant investment in new HRM initiatives through CAPEX
Challenges
Deciding upon the most critical measures (“vital few versus trivial many”)
The financial literacy of L&D (and HR staff more broadly) is highly variable across organisations
Proving links between the training intervention/s and the operating and business outcomes
“how confident am I that the increase in sales was due to training as opposed to advertising/promotional activity”
Opportunities
Develop stronger business cases to secure, maintain resource allocation to L&D
To demonstrate L&D activity as synchronised to operations and overall organisation goals
Better secure significant investment in new HRM initiatives through CAPEX
Challenges
Deciding upon the most critical measures (“vital few versus trivial many”)
The financial literacy of L&D (and HR staff more broadly) is highly variable across organisations
Proving links between the training intervention/s and the operating and business outcomes
“how confident am I that the increase in sales was due to training as opposed to advertising/promotional activity”
Opportunities
Develop stronger business cases to secure, maintain resource allocation to L&D
To demonstrate L&D activity as synchronised to operations and overall organisation goals
Better secure significant investment in new HRM initiatives through CAPEX
Challenges
Deciding upon the most critical measures (“vital few versus trivial many”)
The financial literacy of L&D (and HR staff more broadly) is highly variable across organisations
Proving links between the training intervention/s and the operating and business outcomes
“how confident am I that the increase in sales was due to training as opposed to advertising/promotional activity”
Opportunities
Develop stronger business cases to secure, maintain resource allocation to L&D
To demonstrate L&D activity as synchronised to operations and overall organisation goals
Better secure significant investment in new HRM initiatives through CAPEX
Challenges
Deciding upon the most critical measures (“vital few versus trivial many”)
The financial literacy of L&D (and HR staff more broadly) is highly variable across organisations
Proving links between the training intervention/s and the operating and business outcomes
“how confident am I that the increase in sales was due to training as opposed to advertising/promotional activity”
Opportunities
Develop stronger business cases to secure, maintain resource allocation to L&D
To demonstrate L&D activity as synchronised to operations and overall organisation goals
Better secure significant investment in new HRM initiatives through CAPEX
Challenges
Deciding upon the most critical measures (“vital few versus trivial many”)
The financial literacy of L&D (and HR staff more broadly) is highly variable across organisations
Proving links between the training intervention/s and the operating and business outcomes
“how confident am I that the increase in sales was due to training as opposed to advertising/promotional activity”
Opportunities
Develop stronger business cases to secure, maintain resource allocation to L&D
To demonstrate L&D activity as synchronised to operations and overall organisation goals
Better secure significant investment in new HRM initiatives through CAPEX
Challenges
Deciding upon the most critical measures (“vital few versus trivial many”)
The financial literacy of L&D (and HR staff more broadly) is highly variable across organisations
Proving links between the training intervention/s and the operating and business outcomes
“how confident am I that the increase in sales was due to training as opposed to advertising/promotional activity”
