PD518, Training Evaluation Workshop course slides. U.S. Department of State. Foreign Service Institute.

1. Training Evaluation WorkshopPD518May 12-13, 2011 Facilitator: Kathy Beckman FSI/SPAS/CSD

2. Training Evaluation is a “Hot Topic” because . . .

3. Setting the Learning Space Expectations, Learning Objectives, Group Norms

4. As a result of participating in this workshop, you should be able to: Define training evaluation Differentiate between Kirkpatrick’s four levels of training evaluation Write learning objectives that are specific and measurable Learning Objectives

5. Identify strategies and methods for assessing student reactions to training (Level 1) Specify steps for creating effective, valid tests to measure learning gains (Level 2). Identify strategies and methods for determining changes in behavior after training (Level 3). Locate resources for assessing organizational results (Level 4) Develop an Evaluation Plan for evaluating your training course/program. Learning Objectives, cont.

6. Your Training Evaluation Plan Throughout the course, you will be using templates to develop a Training Evaluation Plan for your own course(s)

7. Training Evaluation Overview How do we know that our training works?

8. How did participants react? Were they satisfied? Did it meet their needs? Would they recommend it to others?

9. What did they learn? Knowledge? Skills? Attitudes? “KSAs”

10. Did behavior (performance) change as a result? How have they applied knowledge, skills, and attitudes back on the job?

11. What is the organizational impact? Did the organization achieve its desired results from the training?

12. Training Evaluation is… “An integrated, four-level approach to determine the effectiveness of training programs.” Source: Kirkpatrick, Donald L. and Kirkpatrick, James D. Evaluating Training Programs. The Four Levels. 3rd Edition. San Francisco: Berrett-Koehler. 2006. 7

13. Training Evaluation is… “A systematic process to determine the worth, value, or meaning of a training activity or process.” Source: Jack J. Phillips and Ron Drew Stone. How to Measure Training Results. A Practical Guide to Tracking the 6 Key Indicators. New York: McGraw-Hill. 2002. 7

14. Why Evaluate? Sources: Kirkpatrick, Donald L. and Kirkpatrick, James D. Evaluating Training Programs. The Four Levels. Jack J. Phillips and Ron Drew Stone. How to Measure Training Results.

15. Successful Training Evaluation is… Aligned with organization’s mission and strategic goals A systematic process Data-driven Focused on continuing improvement

16. Benefits of Successful Training Evaluation Provides data over time (trends) Multiple stakeholders can use data as evidence when making decisions Helps improve quality of training activities Contributes to maximizing the talents of Department of State personnel

17. FSI’s Training Evaluation Model Kirkpatrick’s Four Levels

18. Level 4 Organizational Results  Plan  Evaluate Level 3 Student Behavior Change on the job Level 2 Student Learning Level 1 Student Reaction  Train (KIRKPATRICK MODEL) 9

19. Evaluation Timeframes

20. Key Points about the Kirkpatrick Model A sequence of ways to evaluate training Each level is important and yields valuable data Each level impacts the next level – don’t skip levels! Organizations should strategically select the scope of their evaluation activities “All levels for all training” is usually too costly

21. Evaluation Data Creates a “Chain of Evidence” Source: Jim Kirkpatrick, PhD and Wendy Kayser Kirkpatrick. “The Kirkpatrick Four Levels: A Fresh Look After 50 Years. 1959 – 2009.” Copyright 2009. Kirkpatrick Partners, LLC.

22. Sample Evaluation Strategy Source: Strategy of a large telecommunications company. Jack J. Phillips and Ron Drew Stone. How to Measure Training Results, p. 19.

24. Data collection and administration

25. Data analysis

28. All participants

29. Sample

30. Confidence level (95%)

31. Confidence interval (margin of error)

33. What response rate will give you a “statistically valid” sample?

34. How can you increase your response rate?See FSI 2010 Annual Training Survey, p. 48 Industry research paper – on back table

35. Data Collection Methods

36. Data Collection Types Qualitative Data 1. Describes context, meaning 2. Examples: personal experiences, case studies Quantitative Data 1. Can be expressed and analyzedas a number 2. Examples: demographics, rankings (1-5) Both types of data are important!

37. FSI Survey Tools

38. Applying the Kirkpatrick Model at the Department of State

39. Level 4 Organizational Results  Plan  Evaluate Level 3 Student Behavior Change on the job Level 2 Student Learning Level 1 Student Reaction  Train (KIRKPATRICK MODEL)

40. FSI Mission “The mission of the Foreign Service Institute (FSI) is to develop the men and women our nation requires to fulfill our leadership role in world affairs and to defend U.S. interests.” From FY 2011 Bureau Strategic Plan 13

41. FSI’s 4 Strategic Goals Workforce Meets Priority Diplomatic & Operational Requirements as a Result of FSI Training Global Workforce Can More Widely Access Training Through Distance Learning Technologies From FY 2011 Bureau Strategic Plan

42. FSI’s 4 Strategic Goals, cont. Management Practices Promote Efficiency and Effectiveness Core Training Continues To Fulfill Baseline Requirements and Meet New Challenges/New Skills From FY 2011 Bureau Strategic Plan

43. Excerpt from FSI 2010 Annual Training Survey (Level 4) IMPACT OF FSI TRAINING ON IMPROVED JOB PERFORMANCE - % Agree or Strongly Agree 2010 (Question Added in 2010) 94 % Employee Self-Assessment 96% Supervisor Assessment of Employee(s) 48-51

44. Who can identify critical on-the-job behaviors? Leadership teams Supervisors Subject matter experts Within the Department Outside the Department

45. Knowledge Skills Attitudes “KSAs”

46. Examples of Learning Conditions Location Date/time Format (classroom, DVC, Webinar, distance learning) Facilitators Materials & methods

47. Writing Learning Objectives Make them specific and measureable!

48. What is a Learning Objective? A specific statement of measureable results a student can expect to achieve as a result of training Introduced by: “As a result of participating in this training, you will be able to…”

49. Which one of these is a Learning Objective? A – Identify (underline or circle) all misspelled proper nouns in a 500-word Spanish language news article. B – Play soccer. C - Demonstrate to students the correct way to complete a Leave Request form

50. Characteristics of an Effective Learning Objective Performance Conditions Criteria Source: Robert F. Mager. Preparing Instructional Objectives. Revised 2nd Edition. 1984. Lake Publishing Company. Belmont, CA.

51. Reaction Level 1

52. Level 1 Questions What did participants like about the program? What did they not like? How do they plan to use their training on the job?

53. Evaluation Samples– Level 1 Kirkpatrick Chapter 4 pp. 29-34

54. PD512, Training Design Workshop Workbook Generic evaluation for FSI Distance Learning class Workbook SAIT Instructor Led Training, Post Event Survey Handout FSI Evaluation Samples– Level 1 38-41

55. Eight Tips on Developing Valid Level 1 Evaluation Forms Ken Phillips. Training Today, Fall 2007 (A quarterly magazine published by the Chicagoland Chapter of ASTD) Resources for Level 1

56. Are we on target? I want to learn more about…

57. Level 2 Learning: knowledge, skills, attitudes

58. What did participants learn? What knowledge, skills, or attitudes did they develop or enhance during the course? Level 2 Questions

59. Some Level 2 Evaluation Methods

60. PD 505, Training Tradecraft 10 minute training sessions (recorded) Public Diplomacy “Elevator speech” In-country TV interview (recorded) FSI Examples of Performance Tests Do you use performance tests? If so, please describe.

61. PD505, Training Tradecraft Interactive Presentation Checklist for training and presentation skills FSI Level 2 Evaluation Sample 42

62. Research on Kirkpatrick Levels There is statistical correlation between Levels 1 and 2 A positive learner reaction to training resulted in more learning There is also a statistical correlation between Levels 3 and 4 “When employees consistently perform critical on-the-job behaviors, individual and overall productivity increased”1 1 Research for Kirkpatrick Partners conducted by Sandy Almeida,MD,MPH.

63. Research on Kirkpatrick Levels, cont. However, there is not a statistical correlation between Levels 2 and 3 “Even providing excellent training does not lead to significant transfer of learning to behavior and subsequent results without a good deal of deliberate and consistent reinforcement.”2 2 Ibid., p. 5.

64. Failure to Transfer Learning to the Workplace More than 70% of learning failures occur after the training event is over.

65. Behavior Level 3

66. Level 3 Questions “Rubber meets the road” What specific behavior (performance) changes have resulted from the training? How well are participants applying the KSAs from training?

67. PD505, Training Tradecraft evaluation of 3 regional training classes School of Language Studies, FSI Classroom Training Impact Survey Handout Evaluation Samples– Level 3 43-47

68. Resources for Level 3 Quick Tips – a weekly online newsletter with practical tips on improving learning transfer to behavior on the job Register at www.kirkpatrickpartners.com Kirkpatrick Evaluation group www.linkedin.com

69. Organizational Results Overview of Level 4

70. Level 4 Questions What is the organizational impact? How did individual behavior change result in organizational success? The global view

71. Excerpt from FSI 2010 Annual Training Survey (Level 4) IMPACT OF FSI TRAINING ON IMPROVED JOB PERFORMANCE - % Agree or Strongly Agree 2010 (Question Added in 2010) 94% Employee Self-Assessment 96% Supervisor Assessment of Employee(s) 48-51

72. Resources for Level 4 Books, articles, white papers Donald Kirkpatrick Jim Kirkpatrick Jack J. Phillips Websites American Society for Training and Development Kirkpatrick Partners ROI Institute (Phillips)

73. Complete your Training Evaluation Planning Templates, pp. 14, 33

74. Complete the templates for your training course/program Workbook, p. 14 – Starting your Training Evaluation Plan Workbook, p. 33 Data Collection Plan Worksheet Get together with partner(s) and share your evaluation plan Give and receive feedback to perfect your plans! Learning Activity

75. Presentations of Training Evaluation Plans