Evaluations and feedback
•Transferir como PPTX, PDF•
0 gostou•1,227 visualizações
An introduction to evaluation and feedback. This slide deck covers basic concepts every person engaged with designing evaluation or feedback systems for training should be familiar with. I gave this presentation to MBA students in Iqra University as a guest speaker in 2018. Hopefully, it could be useful for more people.
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Evaluations and feedback
Semelhante a Evaluations and feedback (20)
Mais de Zeeshan Shahid
Mais de Zeeshan Shahid (12)
Último
Último (20)
Evaluations and feedback
- 1. Evaluation and Feedback ZEESHAN SHAHID MARCH 3, 2018
- 2. Citations ASTD Handbook for Workplace Learning Professionals Pros and Cons of Training Methods, by Craig Borysowich The Art and Science of Training The Most Effective Training Techniques Evaluation and Feedback, by Zeeshan Shahid 2 (ed),ElaineBiech.ASTDHandbookfor WorkplaceLearning Professionals.AssociationforTalent Development.©2008.Skillsoft. Biech,Elaine.TheArtandScienceof Training.AssociationforTalent Development.©2017.Skillsoft.
- 3. Evaluation INTRODUCTION TO THE FOUR LEVELS Evaluation and Feedback, by Zeeshan Shahid 3
- 4. Kirkpatrick’s Four levels of evaluation In today's competitive environment, there's no question that we need to evaluate and prove the value of our training programs. That, of course, is easier said than done. Fortunately, the four-level evaluation model simplifies what can be a complex process. Reaction: A measure of the satisfaction of the participants who attended the program Learning: The extent to which participants increased their knowledge, learned or improved present skills, or changed their attitudes Behavior: The extent to which participants applied what they learned when they returned to their jobs Results: The improvement of morale, the increase in sales or production, the reduction in turnover, the increase in customer satisfaction, the return-on-investment (ROI), and any other benefits that came from attending the program. Evaluation and Feedback, by Zeeshan Shahid 4
- 5. Level 1 evaluation REACTION Evaluation and Feedback, by Zeeshan Shahid 5
- 6. Level 1 evaluation: Reactions Typically consists of a reaction sheet The purpose is to measure how the attendees feel about the program. Unfortunately, some people put little faith in this evaluation level and negatively refer to it as smile sheets Without judging the method, lets review the definition of reaction, which measures customer satisfaction Evaluation and Feedback, by Zeeshan Shahid 6
- 7. Guidelines for evaluating reactions Evaluation and Feedback, by Zeeshan Shahid 7
- 8. Level 2 evaluation LEARNING Evaluation and Feedback, by Zeeshan Shahid 8
- 9. Level 2 evaluation: Learning Level 2 comes after Level 1 (in case it wasn’t obvious) Key questions What knowledge was learned? What skills were developed or improved? What attitudes were changed? Why measure learning? Evaluation and Feedback, by Zeeshan Shahid 9
- 10. Guidelines for evaluating learning Measure KSA before and after For K and A: Use paper/pencil test For S: Use performance test Evaluation and Feedback, by Zeeshan Shahid 10 <=100 % responses if possible If practical, use a control group (untrained) and experimental group (trained)
- 11. Data collection methods Evaluation and Feedback, by Zeeshan Shahid 11 Written questionnaires Phone surveys Interviews Focus groups Tests Observations Performance tests Extant Data or Client/Company Performance Records
- 12. Level 3 evaluation BEHAVIOUR Evaluation and Feedback, by Zeeshan Shahid 12
- 13. Level 3 evaluation: Behavior Objective is to answer: What happens when trainees leave the classroom and return to their jobs? How much transfer of knowledge, skills, and attitudes occurs? Complicated than first two levels: No change in behavior can occur unless learner has had the opportunity to do so. Impossible to predict when the change will occur Outcome of applying learning on the job can either be: Liking the outcome and continue to use new behavior Not liking outcome and continue old behavior Liking the outcome but circumstances restricted learner to old behavior We all hope that the rewards for changing behavior will cause the trainee to come to the first of these conclusions. Evaluation and Feedback, by Zeeshan Shahid 13
- 14. Guidelines for evaluating behavior Measure on before and after, if possible If not possible, measure after and ask self assessment of before Allow time for new behavior to be used Evaluation and Feedback, by Zeeshan Shahid 14 100 % responses or a sample Repeat at appropriate interval If practical, use a control group
- 15. Guidelines for evaluating behavior One good way to measure change in behavior is to use a patterned interview with a select group that includes the participants and possibly their supervisors, peers, and even subordinates. In the interview, ask to what extent the participant has changed his or her behavior because of attending the program. Possible answers are To a large extent To some extent Not at all. If the participant says, "Not at all," ask him or her, "Why not?" with the possible answers being The training did not apply to my job. I intended to change but had higher priorities. My boss prevented or discouraged me from changing. I have had no opportunity to use it yet. Be sure when asking these questions that you do not put the participant on the spot by putting pressure for a positive answer. Emphasize that you want the truth so that the program can be evaluated and improved. Evaluation and Feedback, by Zeeshan Shahid 15
- 16. Level 4 evaluation RESULTS Evaluation and Feedback, by Zeeshan Shahid 16
- 17. Level 4 evaluation: Results Finally we get to what its all about: results. It is important to go through all three evaluation levels before trying to evaluate results—and ROI in particular. Results are the last thing to evaluate, but the first thing to consider when planning a training program. Questions to ponder when planning: What results do we hope to achieve through a training program? What behaviors are needed to accomplish those results? What knowledge, skills, and attitudes are necessary to initiate these behaviors? How can I present a training program that is practical, interactive, and enjoyable? Evaluation and Feedback, by Zeeshan Shahid 17
- 18. Guidelines for evaluating behavior Measure on before and after basis Allow time for possible results to take place Repeat at appropriate times Evaluation and Feedback, by Zeeshan Shahid 18 Use a control group, if practical Consider cost vs. benefit Look for evidence, as absolute proof isn’t possible
- 19. Guidelines for evaluating behavior Evaluating results is usually easier than evaluating changes in behavior because specific data are often readily available. For example, when measuring the reduction in turnover, figures are easy to get. Likewise, increase in sales, increase or retention of customers, change in profits, amount of scrap, and time to complete a job are usually available. Soft skills such as leadership, communication, managing change, and decision making are more difficult to measure and cannot be measured in dollars or in ROI. Some specific measures are available that can relate to these programs if the organization has an annual attitude survey, but converting the comparisons to money is not viable unless they can be converted to such measurable items as turnover. Note that a "chain of evidence" can be created by evaluating all four of the levels in sequence. Each bit of evidence can help build the overall case that the program was effective. So, when evaluating, do all four levels and don't yield to the temptation of jumping from learning to results or ROI. Evaluation and Feedback, by Zeeshan Shahid 19
- 20. ” “Although learning is the desired goal of any training program, the most critical aspect is that the learning creates the desired change in behavior that must take place for positive results to be achieved. DONALD KIRKPATRICK Evaluation and Feedback, by Zeeshan Shahid 20
- 21. Feedback OVERVIEW Evaluation and Feedback, by Zeeshan Shahid 21
- 22. Seven principles of good feedback practices (HEA 2004) 1. Facilitates the development of self-assessment (reflection) in learning. 2. Encourages teacher and peer dialogue around learning. 3. Helps clarify what good performance is (goals, criteria, standards expected). 4. Provides opportunities to close the gap between current and desired performance. 5. Delivers high quality information to students about their learning. 6. Encourages positive motivational beliefs and self-esteem. 7. Provides information to teachers that can be used to help shape the teaching. Evaluation and Feedback, by Zeeshan Shahid 22
- 23. Feedback best practices Quick feedback can be done using a Feedback Sandwich: Evaluation and Feedback, by Zeeshan Shahid 23 Praise Identify something that went well Feedback Then suggest something that would benefit from change Praise Then reinforce the positives
- 24. Feedback best practices More detailed feedback can follow Pendleton’s Rules: Evaluation and Feedback, by Zeeshan Shahid 24 Briefly clarify matters of fact The learner goes first and discusses what went well The trainer discusses what went well The learner describes what could have been done differently and makes suggestions for change The trainer identifies what could be done differently and gives options for change
- 25. Feedback best practices BOOST is a popular model, relevant for both praise and constructive feedback. It keeps things positive, while still ensuring that any problem areas are addressed, by helping you ensure that feedback is: Evaluation and Feedback, by Zeeshan Shahid 25 Balance d Observe d Objectiv e Specific Timely
- 26. Feedback best practices Evaluation and Feedback, by Zeeshan Shahid 26
- 27. Feedback best practices Evaluation and Feedback, by Zeeshan Shahid 27
- 28. Questions Evaluation and Feedback, by Zeeshan Shahid 28
Notas do Editor
- Customer satisfaction is essential. If customers report bad things on a reaction sheet, they likely will say bad things when they return to their job. The result: Higher management may judge the training based on what they hear. Therefore, reactions are important to evaluate, not only to benefit the instructor, but to know that customers have gone back to their jobs with a positive attitude toward the program and the trainers who presented it. Additionally, if you don't measure customer reaction and satisfaction, you are subtly telling them that their feedback isn't wanted or needed. The answer is simple: use reaction sheets for all programs.
- Why do we measure learning? Simply put, if the training course does not meet one or more of the learning objectives, you cannot expect any change in behavior. Moreover, if we were to measure behavior change (level 3) and not learning, and if we found no change in behavior, the likely conclusion would be that no learning took place. This conclusion could be very wrong. Evaluating learning is important for two reasons. First, it measures the effectiveness of the instructor in increasing knowledge and changing attitudes. If little or no learning has taken place, little or no change in behavior can be expected. Just as important is the specific information that pretest and posttest evaluation of learning provides. By analyzing the change in answers to individual items, instructors can see where they have succeeded and where they have failed. If the program is going to be repeated, instructors can plan other techniques or aids to increase the chances that learning will take place. Moreover, if follow-up sessions can be held with the same group, the things that have not been learned can become the objectives of these sessions.
- This level is more complicated than the first two levels. First, trainees cannot change their behavior until they have an opportunity to do so. For example, if you decide to use some of the principles and techniques described here, you must wait until you have a training program to evaluate. Likewise, if the training program is designed to teach a person how to conduct an effective performance appraisal interview, the trainee cannot apply the learning until an interview is held. Second, it is impossible to predict when a change in behavior will occur. Even if a trainee has an opportunity to apply the learning, he or she may not do so immediately. In fact, behavior changes may occur at any time after the first opportunity, or they may never occur.
- Finally we get to what its all about: results. It is important to go through all three evaluation levels before trying to evaluate results—and ROI in particular. Results are the last thing to evaluate, but the first thing to consider when planning a training program. When planning a program, the questions to ask, in order, are What results do we hope to achieve through a training program? What behaviors are needed to accomplish those results? What knowledge, skills, and attitudes are necessary to initiate these behaviors? How can I present a training program that is practical, interactive, and enjoyable? Some guidelines for evaluating results include Measure on a before-and-after basis. Allow time for possible results to take place. Repeat at appropriate times. Use a control group if practical. Consider costs versus benefits. Look for "evidence" because absolute "proof" is not possible. Evaluating results is usually easier than evaluating changes in behavior because specific data are often readily available. For example, when measuring the reduction in turnover, figures are easy to get. Likewise, increase in sales, increase or retention of customers, change in profits, amount of scrap, and time to complete a job are usually available. Soft skills such as leadership, communication, managing change, and decision making are more difficult to measure and cannot be measured in dollars or in ROI. Some specific measures are available that can relate to these programs if the organization has an annual attitude survey, but converting the comparisons to money is not viable unless they can be converted to such measurable items as turnover. Note that a "chain of evidence" can be created by evaluating all four of the levels in sequence. Each bit of evidence can help build the overall case that the program was effective. So, when evaluating, do all four levels and don't yield to the temptation of jumping from learning to results or ROI.