This document summarizes a knowledge sharing session on measuring learning impact. The session: 1) Discussed the importance of learning and development programs in organizations and different models for evaluating training, including Kirkpatrick's four levels of evaluation. 2) Presented two sample training courses and asked how evaluation could be conducted for each, focusing on which level of evaluation is most appropriate. 3) Noted that for the sample courses, level 4 evaluation measuring business impact would not be possible due to challenges in collecting relevant data. 4) Emphasized that the purpose of learning interventions is to impact the business through revenue generation, cost reduction, process improvement and people engagement.

1. Knowledge Sharing Session
Measuring Learning Impact
1
Facilitator: Noor Iskandar
28 May 2021

2. LET’S START BY PUTTING
THINGS IN PERSPECTIVE

3. The Importance Of Learning And Development In An Ever-
Changing World
• When looking at how to keep an organization in top form, learning and development can be one
of the best ways. The Association for Talent Development, based in the United States, concludes
that “organizations are more competitive, agile, and engaged when knowledge is constantly and
freely shared.”
• In fact, after surveying around 830 companies in 2015, they found that organizations with a
culture of learning were some of the highest performers. They also found that they attracted more
talented employees and had higher levels of customer satisfaction.
• Research done through LinkedIn found that the main way for employees to be productive and
achieve goals is for employers to provide them with opportunities for continuous learning. This
helps employees continue to improve their skills and be more flexible and adaptable, as changes
related to technology and other workplace matters can arise quickly. When continuous learning is
supported employees feel valued, helping boost employee engagement and retention.
• Seeing how beneficial learning and development programs are, why is it that only 31% of
organizations surveyed had well developed learning and development programs?
3
https://gethppy.com/talent-management/importance-of-learning-and-development extracted on 24 May 2021

4. ROLE OF LEARNING AND
DEVELOPMENT IN AN ORGANIZATION
The Learning & Development Manager is responsible for developing and implementing an organization’s learning strategy.
Some but not limited are as below:
• Develop and implement learning strategies and programs that meet business needs
• Evaluate organizational as well as individual employee development needs
• Ensure alignment of learning & development department with business goals
• Optimize training processes to improve ROI
• Manage and procure L&D technologies to facilitate learning and management of learning initiatives
• Design eLearning courses, interactive group training, and other training content
• Implementing effective training methods
• Manage personnel within the learning & development department
• Maintaining the learning & development budget within an organization
• Championing and marketing training programs to get managerial buy-in as well as employee commitment
The learning & development manager is central to ensuring that employees are able to successfully meet the challenges
of their jobs and that they are properly aligned with the business goals of the company.

5. 5
Importance
of impact of
L&D and
measuring
them.

6. 6
Why the need to Evaluate Learning & Development
Interventions

7. MARCH/APRIL 2020 training www.trainingmag.com
Business leaders’ thoughts and perceptions of L&D
Mind Tools for Business survey report, March 2021
The Needs Analysis Playbook: How to Make L&D a Trusted Partner in
Your Organization
©2021 All rights reserved. www.sweetrush.com

8. L&D is necessary in developing not just Capability,
also Maturity

9. First Edition, Jan 1990
Learning
Organization

10. Last but not least,
something very recent
2020
2021

11. Bottom Line…
https://www.hrdconnect.com
/2020/01/28/grow-your-
people-and-you-will-grow-
your-business/
Extracted on 25 May 2021

12. Measuring Learning Impact
begins with the
Design of the Learning Solutions
12

13. Effective Training
Effective training is considered as an important factor in determining the
efficiency of an organization which depends upon the capability of its
employees.
For training evaluation to be truly effective, the training and development
itself must be appropriate for the person and the situation.
Even though evaluation is listed at the last phase, evaluation actually
happens during all the phases. It is used during the training process to
evaluate the training process itself. Evaluation is not just for the trainer or
organization it is absolutely important for the learner too.
One new model was found namely Back planning model. Which is same
as Kirkpatrick’s model but the process occurs in the reverse order and
continues in a cyclic process.
Analysis and Evaluation of Training Effectiveness
Akhila Kunche , Ravi Kumar Puli , Sunitha Guniganti , Danaiah Puli
N.I.T. Warangal Andhra Pradesh, India

14. 14

15. 15

16. 2 Commonly Used Models in
Developing Training
16

17. A-D-D-I-E: Learning Development Process

18. 6Ds Approach to Learning Development Process
18

19. Good Learning Occurs…
Effective
Delivery
Contextual
Relevancy
Content
Validity
Hence, learning evaluation will be
based on all of them

20. Measuring
Learning
Impact
20

21. Before we measure learning impact,
let’s get to know some fundamentals
21

22. Two technical concepts
Testing (Measuring),
Evaluation &
Assessment
Reliability and
Validity

23. Testing (Measuring), Evaluation & Assessment
In case you are curious, here are some definitions:
• A test or quiz is used to examine someone's knowledge of something to
determine what he or she knows or has learned. Testing measures the
level of skill or knowledge that has been reached.
• Evaluation is the process of making judgments based on criteria and
evidence.
• Assessment is the process of documenting knowledge, skills, attitudes
and beliefs, usually in measurable terms. The goal of assessment is to
make improvements, as opposed to simply being judged. In an
educational context, assessment is the process of describing, collecting,
recording, scoring, and interpreting information about learning.

24. Reliability and Validity
• These two terms, reliability and validity, are often used
interchangeably when they are not related to statistics. When critical
readers of statistics use these terms, however, they refer to different
properties of the statistical or experimental method.
• Reliability is another term for consistency. If one person takes the same
personality test several times and always receives the same results, the
test is reliable.
• A test is valid if it measures what it is supposed to measure. If the results
of the personality test claimed that a very shy person was in fact
outgoing, the test would be invalid.
• Reliability and validity are independent of each other. A measurement
maybe valid but not reliable, or reliable but not valid. Suppose your
bathroom scale was reset to read 10 pound lighter. The weight it reads
will be reliable (the same every time you step on it) but will not be valid,
since it is not reading your actual weight.

25. Training Evaluation Models
25

26. What are the commonly available
training evaluation models?
The
Kirkpatrick
Model
Phillip’s
ROI Model
The CIPP
Model
The CIRO
Model
The
Brinkerhoff
Model
Kaufman's
Model of
Learning
Evaluation
Anderson
Model of
Learning
Evaluation
26
1 2 3 4 5 6 7

27. 27

28. 28
Developed by
Dr. Jack Phillips
Kirkpatrick’s
FOUR Levels

29. Purpose of Learning and Training vis a vis Evaluation
29
4
3
2
1

30. Types of Evaluation Employed by Organizations
30

31. Develop Tests – from
example Objectives below
Learning Objectives
Given the ABC system, display the
customer file in 30 seconds and on
the first try.
Faced with a fire, select
appropriate procedure for
combating it.
Name the capitals of all provinces
in Canada, matching each capital
to its provinces with no errors.
From a collection of bones,
identify the femur.
Test Suggestion
Performance Test. Give the learner a customer name. Have
the learner interact with the ABC system to display the file.
Check for accuracy, time and number of attempts.
Recall Test. Show different types of fires (eg Electrical, gas,
flammable liquid). Have learner select the procedure from
memory. (We have not asked him/her to combat the fire,
only to select the procedure.) Verify accuracy of match.
Recall Test. Have learners recite the provinces and their
capitals from memory. Verify accuracy.
Recognition Test. Display bones. Name femur. Have learner
point to it.

32. Reading Article
Article 1

33. Let’s Focus on Kirkpatrick’s
Levels of Training Evaluation
33

34. Donald Kirkpatrick
• Kirkpatrick developed a model of
training evaluation in 1959.
• Arguably the most widely used
approach.
• Simple, Flexible and Complete
• 4-level model.

35. Why Evaluate?
• Should the program be continued?
• How can the program be improved?
• How can we ensure regulatory compliance?
• How can we maximize training effectiveness?
• How can we be sure training is aligned with
strategy?
• How can we demonstrate the value of training?

36. The Four Levels of Evaluation
• Level I: Evaluate Reaction
• Level II: Evaluate Learning
• Level III: Evaluate Behavior
• Level IV: Evaluate Results
• Fifth level was recently “added” for return on
investment (“ROI”) but this was not in
Kirkpatrick’s original model

37. Relationship Between Levels
• Each subsequent level is
predicated upon doing
evaluation at lower level
• A Level 3 will be of marginal
use, if a Level 2 evaluation is
not conducted
Level 1 - Reaction
Was the environment suitable for learning?
Level 2 - Knowledge
Did they learn anything
Level 3 - Behavior
KSA being used on the job?
Level 4 - Results
Was it worth it?

38. Slide 38
Improve
Learning Environment
Improve
Knowledge/Skill transfer
Check
Performance Environment
Check Requirements,
Systems and Processes
Only by assessing each level can we yield actionable results
Level 1 - Reaction
Was the environment suitable for learning?
Level 2 - Knowledge
Did they learn anything
Level 3 - Behavior
KSA being used on the job?
Level 4 - Results
Was it worth it?

39. Types of Assessments Used at Each Level
Level 1 - Reaction
Was the environment suitable for learning?
Level 2 - Knowledge
Did they learn anything
Level 3 - Behavior
KSA being used on the job?
Level 4 - Results
Was it worth it?
Type Form
Summative
Correlation of business results
with other assessment results
Summative Observation of Performance
360° Survey
Diagnostic
Summative
Self-assessment
Test
Reaction
Formative
Survey
Real-time Polling
Quizzing

40. Summary of Tools to Purpose
If you want to… Level 1 Level 2 Level 3 Level 4
Continue program X X
Improve program X X X
Ensure compliance X
Maximize training effectiveness X
Align training with strategy X X
Demonstrate training’s value X X X X

41. Evaluation Techniques
Evaluation Levels
Methods
1
Reaction
2
Learning
3
Behavior
4
Results
Survey ● ● ● ●
Questionnaire/Interview ● ● ● ●
Focus Group ● ● ● ●
Knowledge Test/Check ●
Skills Observation ● ●
Presentations ● ●
Action Planning ● ● ●
Action Learning ●
Key Business HR Metrics ●

42. Let’s do a couple of exercises
to see if we know this stuff…
Before we do that…jeng jeng jeng
42

43. Sample #1: Plant Operation and Maintenance Training
COURSE OBJECTIVES
1. Describe in detail how to read P & I diagrams and the wealth of information they contain
2. Explore the main challenges of plant start-up, continuous operation & optimization, and shutdown
3. Equip participants with the ability to operate plants efficiently and quantify plant operations through
performance indicators and technical calculations
4. Develop participants’ capacity for troubleshooting methodologically using available process trends
and events recording
5. Present real case scenarios and examples for materializing the
troubleshooting process and focusing on the importance of root causes
6. Investigate plant maintenance strategies and priorities
7. Empower participants to plan turnarounds, execute, and evaluate them.
8. Emphasize the importance and relevance of core operations values and
safety principles on operations processes
9. Explain the creation and implementation of SOP
• How would you see the
evaluation look like?
• Which level of
Evaluation should you
employ?
• How do you think the
training should be
conducted?

44. Sample #2: Management Training for the New Manager or Supervisor
• Including how to:
1. Build a successful team.
2. Prioritize new management responsibilities.
3. Effectively communicate with new staff.
• Participants will learn to:
1. Differentiate between “managing” and “leading”.
2. Effectively transition from staff to management.
3. Establish performance standards and expectations for new teams.
4. Create action plans to reach goals.
5. Delegate effectively.
6. Deliver effective feedback, both positive and negative.
7. Facilitate conflict resolution among staff.
8. Understand the work style differences among different generations.
9. Establish trust and credibility while building team culture.
• How would you see
the evaluation look
like?
• Which level of
Evaluation should
you employ?
• How do you think
the training should
be conducted?

45. Based on the two public courses
examples, you will notice there is no
Level 4 – Impact Evaluation
Why?
45

46. Purpose of Learning and Training vis a vis Evaluation
46

47. To Summarize this section…
Revenue Generation
Cost
Reduction
Process Improvement
People
Engagement
Purpose of Learning
Interventions are to
Impact the Business for:

48. ROI Model and Process
How to Measure Training Results : A Practical Guide to
Tracking the Six Key Indicators Hardcover – February 27, 2002
by Jack Phillips (Author), Ron Stone (Author)
Many organizations have challenges in collecting
and extracting data. Hence, conducting ROI for
training will be difficult.

49. Summary
Analysis
•What
Knowledge?
(L2)
•What Skills or
Behavior?
(L3)
•What
Business
Impact? (L4)
•Is your
training
worth the
investment?
(L5)
Design
•Learning
Outcome to
solve issues in
Analysis.
•What test or
measurement do
we use to
determine if
Learning
Outcome is
achieved?
Develop
•Training &
Learning
Materials to
support
Analysis and
Design,
including the
test /
evaluation.
Implement
(Delivery)
•What delivery
style or
approach will
be able to
meet learning
outcome and
the test /
evaluation?
Evaluate
•Actual
evaluation
happens
49

50. Finally, looking at ADDIE model
Analyze
Design
Develop
Implement
Evaluate
Analyze
Design
Develop
Implement Evaluate
Delivery
Areas L&D should be good at

51. Again…
51
4
3
2
1

52. We are done for this session…
52

53. Knowledge Sharing Session
Measuring Learning Impact
53
Thank you