4. As a result of participating in this workshop, you should be able to: Define training evaluation Differentiate between Kirkpatrick’s four levels of training evaluation Write learning objectives that are specific and measurable Learning Objectives
5. Identify strategies and methods for assessing student reactions to training (Level 1) Specify steps for creating effective, valid tests to measure learning gains (Level 2). Identify strategies and methods for determining changes in behavior after training (Level 3). Locate resources for assessing organizational results (Level 4) Develop an Evaluation Plan for evaluating your training course/program. Learning Objectives, cont.
6. Your Training Evaluation Plan Throughout the course, you will be using templates to develop a Training Evaluation Plan for your own course(s)
8. How did participants react? Were they satisfied? Did it meet their needs? Would they recommend it to others?
9. What did they learn? Knowledge? Skills? Attitudes? “KSAs”
10. Did behavior (performance) change as a result? How have they applied knowledge, skills, and attitudes back on the job?
11. What is the organizational impact? Did the organization achieve its desired results from the training?
12. Training Evaluation is… “An integrated, four-level approach to determine the effectiveness of training programs.” Source: Kirkpatrick, Donald L. and Kirkpatrick, James D. Evaluating Training Programs. The Four Levels. 3rd Edition. San Francisco: Berrett-Koehler. 2006. 7
13. Training Evaluation is… “A systematic process to determine the worth, value, or meaning of a training activity or process.” Source: Jack J. Phillips and Ron Drew Stone. How to Measure Training Results. A Practical Guide to Tracking the 6 Key Indicators. New York: McGraw-Hill. 2002. 7
14. Why Evaluate? Sources: Kirkpatrick, Donald L. and Kirkpatrick, James D. Evaluating Training Programs. The Four Levels. Jack J. Phillips and Ron Drew Stone. How to Measure Training Results.
15. Successful Training Evaluation is… Aligned with organization’s mission and strategic goals A systematic process Data-driven Focused on continuing improvement
16. Benefits of Successful Training Evaluation Provides data over time (trends) Multiple stakeholders can use data as evidence when making decisions Helps improve quality of training activities Contributes to maximizing the talents of Department of State personnel
20. Key Points about the Kirkpatrick Model A sequence of ways to evaluate training Each level is important and yields valuable data Each level impacts the next level – don’t skip levels! Organizations should strategically select the scope of their evaluation activities “All levels for all training” is usually too costly
21. Evaluation Data Creates a “Chain of Evidence” Source: Jim Kirkpatrick, PhD and Wendy Kayser Kirkpatrick. “The Kirkpatrick Four Levels: A Fresh Look After 50 Years. 1959 – 2009.” Copyright 2009. Kirkpatrick Partners, LLC.
22. Sample Evaluation Strategy Source: Strategy of a large telecommunications company. Jack J. Phillips and Ron Drew Stone. How to Measure Training Results, p. 19.
36. Data Collection Types Qualitative Data 1. Describes context, meaning 2. Examples: personal experiences, case studies Quantitative Data 1. Can be expressed and analyzedas a number 2. Examples: demographics, rankings (1-5) Both types of data are important!
39. Level 4 Organizational Results Plan Evaluate Level 3 Student Behavior Change on the job Level 2 Student Learning Level 1 Student Reaction Train (KIRKPATRICK MODEL)
40. FSI Mission “The mission of the Foreign Service Institute (FSI) is to develop the men and women our nation requires to fulfill our leadership role in world affairs and to defend U.S. interests.” From FY 2011 Bureau Strategic Plan 13
41. FSI’s 4 Strategic Goals Workforce Meets Priority Diplomatic & Operational Requirements as a Result of FSI Training Global Workforce Can More Widely Access Training Through Distance Learning Technologies From FY 2011 Bureau Strategic Plan
42. FSI’s 4 Strategic Goals, cont. Management Practices Promote Efficiency and Effectiveness Core Training Continues To Fulfill Baseline Requirements and Meet New Challenges/New Skills From FY 2011 Bureau Strategic Plan
43. Excerpt from FSI 2010 Annual Training Survey (Level 4) IMPACT OF FSI TRAINING ON IMPROVED JOB PERFORMANCE - % Agree or Strongly Agree 2010 (Question Added in 2010) 94 % Employee Self-Assessment 96% Supervisor Assessment of Employee(s) 48-51
44. Who can identify critical on-the-job behaviors? Leadership teams Supervisors Subject matter experts Within the Department Outside the Department
48. What is a Learning Objective? A specific statement of measureable results a student can expect to achieve as a result of training Introduced by: “As a result of participating in this training, you will be able to…”
49. Which one of these is a Learning Objective? A – Identify (underline or circle) all misspelled proper nouns in a 500-word Spanish language news article. B – Play soccer. C - Demonstrate to students the correct way to complete a Leave Request form
50. Characteristics of an Effective Learning Objective Performance Conditions Criteria Source: Robert F. Mager. Preparing Instructional Objectives. Revised 2nd Edition. 1984. Lake Publishing Company. Belmont, CA.
54. PD512, Training Design Workshop Workbook Generic evaluation for FSI Distance Learning class Workbook SAIT Instructor Led Training, Post Event Survey Handout FSI Evaluation Samples– Level 1 38-41
55. Eight Tips on Developing Valid Level 1 Evaluation Forms Ken Phillips. Training Today, Fall 2007 (A quarterly magazine published by the Chicagoland Chapter of ASTD) Resources for Level 1
60. PD 505, Training Tradecraft 10 minute training sessions (recorded) Public Diplomacy “Elevator speech” In-country TV interview (recorded) FSI Examples of Performance Tests Do you use performance tests? If so, please describe.
61. PD505, Training Tradecraft Interactive Presentation Checklist for training and presentation skills FSI Level 2 Evaluation Sample 42
62. Research on Kirkpatrick Levels There is statistical correlation between Levels 1 and 2 A positive learner reaction to training resulted in more learning There is also a statistical correlation between Levels 3 and 4 “When employees consistently perform critical on-the-job behaviors, individual and overall productivity increased”1 1 Research for Kirkpatrick Partners conducted by Sandy Almeida,MD,MPH.
63. Research on Kirkpatrick Levels, cont. However, there is not a statistical correlation between Levels 2 and 3 “Even providing excellent training does not lead to significant transfer of learning to behavior and subsequent results without a good deal of deliberate and consistent reinforcement.”2 2 Ibid., p. 5.
64. Failure to Transfer Learning to the Workplace More than 70% of learning failures occur after the training event is over.
66. Level 3 Questions “Rubber meets the road” What specific behavior (performance) changes have resulted from the training? How well are participants applying the KSAs from training?
67. PD505, Training Tradecraft evaluation of 3 regional training classes School of Language Studies, FSI Classroom Training Impact Survey Handout Evaluation Samples– Level 3 43-47
68. Resources for Level 3 Quick Tips – a weekly online newsletter with practical tips on improving learning transfer to behavior on the job Register at www.kirkpatrickpartners.com Kirkpatrick Evaluation group www.linkedin.com
70. Level 4 Questions What is the organizational impact? How did individual behavior change result in organizational success? The global view
71. Excerpt from FSI 2010 Annual Training Survey (Level 4) IMPACT OF FSI TRAINING ON IMPROVED JOB PERFORMANCE - % Agree or Strongly Agree 2010 (Question Added in 2010) 94% Employee Self-Assessment 96% Supervisor Assessment of Employee(s) 48-51
72. Resources for Level 4 Books, articles, white papers Donald Kirkpatrick Jim Kirkpatrick Jack J. Phillips Websites American Society for Training and Development Kirkpatrick Partners ROI Institute (Phillips)
74. Complete the templates for your training course/program Workbook, p. 14 – Starting your Training Evaluation Plan Workbook, p. 33 Data Collection Plan Worksheet Get together with partner(s) and share your evaluation plan Give and receive feedback to perfect your plans! Learning Activity
To improve current and future training courses/programsTo decide whether to create, , continue or discontinue training courses/programsTo determine training’s contribution to desired organizational resultsTo maintain an organizationaldatabase for institutional knowledge and future decision-makingWhat do you think of these reasons? How do they relate to your organization?Can you think of additional reasons?
SystematicPlannedDocumentedTrainedImplementedResults communicatedActions taken“How we do business”
Question: Does this look familiar?Each level helps us answer those 4 key questions about how we know that our training works.
Staff Resources and TimeEvaluation planningData collection and administrationData analysis ReportingToolsIn-house and commercial Tool expertise/training
Tips to increase response rate:Communicate Before – let people know in class that you will be following up 3-6 months out Alert – survey email – invite to participate. Follow up – subsequent emails
Tips to increase response rate:Communicate Before – let people know in class that you will be following up 3-6 months out Alert – survey email – invite to participate. Follow up – subsequent emails
Quantitative and Qualitative Data Some methods provide data which are quantitative and some methods data which are qualitative. Quantitative methods are those which focus on numbers and frequencies rather than on meaning and experience. Quantitative methods (e.g. experiments, questionnaires and psychometric tests) provide information which is easy to analyse statistically and fairly reliable. Quantitative methods are associated with the scientific and experimental approach and are criticised for not providing an in depth description. Qualitative methods are ways of collecting data which are concerned with describing meaning, rather than with drawing statistical inferences. What qualitative methods (e.g. case studies and interviews) lose on reliability they gain in terms of validity. They provide a more in depth and rich description. Quantitative methods have come under considerable criticism. In modern research, most psychologists tend to adopt a combination of qualitative and quantitative approaches, which allow statistically reliable information obtained from numerical measurement to be backed up by and enriched by information about the research participants' explanations. You will find that many of the core studies do collect both types of data. http://www.holah.karoo.net/quantitativequalitative.htm
Question: Does this look familiar?Each level helps us answer those 4 key questions about how we know that our training works.
Strategic Goal 1 – “Workforce Meets Priority Diplomatic & Operational Requirements as a Result of FSI Training”Strategic Goal 4 – “Core Training Continues To Fulfill Baseline Requirements and Meet New Challenges/New Skills”
Strategic Goal 1 – “Workforce Meets Priority Diplomatic & Operational Requirements as a Result of FSI Training”Strategic Goal 4 – “Core Training Continues To Fulfill Baseline Requirements and Meet New Challenges/New Skills”
Strategic Goal 1 – “Workforce Meets Priority Diplomatic & Operational Requirements as a Result of FSI Training”Strategic Goal 4 – “Core Training Continues To Fulfill Baseline Requirements and Meet New Challenges/New Skills”
Standard language for training:As a result of training, students will be able to achieve each learning objective
Standard language for training:As a result of training, students will be able to achieve each learning objective
Participant self-assessment Skills in presenting Progress Feedback from others Coaching using a checklist PD505 examples
PD 505, Training Tradecraft and PD513, Training and Presentation Skills10 minute recorded training presentationsPublic Diplomacy“Elevator speech” to a host national walking down the hallPD611, Leading a Small PostSimulated press interview with “reporter” in FSI TV studio
Reported in Kirkpatrick, Jim, PhD and Kirkpatrick, Wendy Kayser. “The Kirkpatrick Four Levels: A Fresh Look after 50 Years.” p. 5.