SlideShare uma empresa Scribd logo
1 de 105
Workforce Development
                    Evaluation




                                 Guide



Version:            2012.03.19
Last released on:   17/04/2012
Last released by:   Dain Sanyë
                    Senior Consultant Evaluation and Continuous Improvement
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




Document title
Evaluation Guide

Document information
This document was produced for the Government of South Australia (SA),
Department for Education and Child Development (DECD), Human Resources and
Workforce Development.
This document was created by Dain Sanyë, Senior Consultant, Evaluation and
Continuous Improvement for Workforce Development, using Microsoft Word 2010.

Purpose
This guide describes the evaluation framework created for the Department’s
Workforce Development directorate.

Audience
This document is designed as a guide for Workforce Development officers and
managers to effectively evaluate and report on projects and programs.
Evaluators should be familiar with:
        the layout of a personal computer and desktop
        the purpose, creation and analysis of data and survey tools
        the Department’s Improvement and Accountability framework (DIAf)
This guide is written in international English.

Contact details
Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement
Phone:            (08) 820 41402
Fax:              (08) 8206 4200
Email:            Dain.Sanye3@sa.gov.au




Copyright notice
Copyright ©2012 Government of South Australia, Department for Education and
Child Development. This publication is copyright and contains information which is
the property of the Department for Education and Child Development. No part of this
document may be copied or stored in a retrieval system without the written
permission of the author or the Chief Executive of the Department for Education and
Child Development.


                                                                                       Page EV–2 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




Related files
        Checkbox survey tool < http://www.decssurveys.sa.edu.au/online/ >
        DIAf < http://www.decd.sa.gov.au/quality/pages/quality/26420/ >

Update plan
Updates are the responsibility of the current Workforce Development Senior
Consultant, Evaluation and Continuous Improvement.
Updates must occur when scheduled as a part of ongoing role responsibility or as
requested by a Workforce Development line manager.
Updates must include an incrementally increased version number in the format
yyyy.mm.dd.A (year.month.date.draft version of update).
Old versions must be moved to an archive location.
The current version must maintain the same file name and be stored and accessible
from a single fixed network location.
Shortcuts should be created at additional locations linking to the single fixed network
storage location.

Revision history
Please destroy any printed copies of this document earlier than version 2012.03.19.
Version           Reviser         Details
2012.02.22.A      Dain Sanyë      Initial draft
2012.03.19        Dain Sanyë      Initial document released




                                                                                       Page EV–3 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement                          Version 2012.03.19




Contents
      Document title ..................................................................................................... 2
      Document information ......................................................................................... 2
      Purpose ............................................................................................................... 2
      Audience ............................................................................................................. 2
      Contact details ..................................................................................................... 2
      Copyright notice................................................................................................... 2
      Related files......................................................................................................... 3
      Update plan ......................................................................................................... 3
      Revision history ................................................................................................... 3
Contents .................................................................................................................. 4
1.0        Introduction.................................................................................................. 7
2.0        Background.................................................................................................. 8
3.0       Evaluative review of a workgroup or project team .................................... 9
      DIAf corporate self-review snapshot survey ......................................................... 9
      Improvement principles...................................................................................... 10
      Improvement principle criteria ............................................................................ 10
          Focus on core business ............................................................................... 10
          Think systemically........................................................................................ 11
          Share leadership.......................................................................................... 11
          Attend to culture .......................................................................................... 11
          Listen and respond ...................................................................................... 11
          Make data count .......................................................................................... 12
          Set directions ............................................................................................... 12
          Target resources.......................................................................................... 12
          Continuously improve .................................................................................. 12
      Principle analysis ............................................................................................... 13
          Aggregate score .......................................................................................... 13
          Graphing foci scores .................................................................................... 14
      Collaborative discussion .................................................................................... 15
3.A        Appendix: Example—DIAf corporate self review survey ........................ 16
4.0       Strategic evaluation of a project ............................................................... 32
      Objectives.......................................................................................................... 33
      Deliverables....................................................................................................... 34
      Strategic plans and alignment............................................................................ 35
      Evaluation measures ......................................................................................... 36
      Focus areas....................................................................................................... 37
          Focusing on development or improvement .................................................. 37
          Designed and implemented with systemic thinking ...................................... 37
          Enables individual leadership....................................................................... 37
          Attentive to organisational culture while transforming capacity..................... 37
          Managed stakeholders through effective listening and responsiveness ....... 37
          Effectively collated, used and reported data................................................. 37


                                                                                              Page EV–4 of EV–105
                                                                                                  Evaluation—Guide
                                                                           R:HRWorkforce DevelopmentProjects and
                                                    InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement                          Version 2012.03.19



         Maintained direction and scope throughout the project ................................ 37
         Innovatively and effectively aligned resources ............................................. 38
         Building a sustainable future including continuous improvement.................. 38
      Scoring .............................................................................................................. 38
         Undeveloped ............................................................................................... 38
         Developing................................................................................................... 38
         Functioning .................................................................................................. 38
         Strategic ...................................................................................................... 38
         Embedded ................................................................................................... 38
4.A         Appendix: Example—strategic project evaluation .................................. 39
5.0      Evaluation of an ongoing program, with a focus on knowledge and skills
           transfer .................................................................................................... 48
      Level 1—evaluation of motivation and reaction .................................................. 49
      Level 2—evaluation of learning content ............................................................. 50
      Level 3—evaluation of performance and behaviour ........................................... 51
      Level 4—evaluation of impact and results ......................................................... 52
A.0       Appendix: Level 1 sample survey questions ........................................... 54
      A.1   Level 1—About the training needs analysis............................................ 54
      A.2   Level 1—About the training .................................................................... 55
      A.3   Level 1—About the participant ............................................................... 56
      A.4   Level 1—Individual learning needs......................................................... 59
      A.5   Level 1—Business needs ....................................................................... 62
      A.6   Level 1—Core skill needs ...................................................................... 64
B.0       Appendix: Level 2 sample survey questions ........................................... 68
      B.1   Level 2—Participant reaction and learning outcomes introduction.......... 68
      B.2   Level 2—General feedback .................................................................... 69
      B.3   Level 2—Training methods and materials .............................................. 70
      B.4   Level 2—Trainer(s) ................................................................................ 72
      B.5   Level 2—Tests and qualifications ........................................................... 73
      B.6   Level 2—Progress to other learning ....................................................... 73
      B.7   Level 2—Pre-training activities and instructions ..................................... 74
      B.8   Level 2—Facilities, courses and resources ............................................ 75
      B.9   Level 2—Facilities and administration .................................................... 76
C.0       Appendix: Level 3 sample survey questions ........................................... 77
      C.1   Level 3—Job performance impact .......................................................... 77
      C.2   Level 3—Relevance of the training......................................................... 78
      C.3   Level 3—Application of learning ............................................................. 80
      C.4   Level 3—Post-training skills observation ................................................ 84
      C.5   Level 3—Core skills improvement .......................................................... 86
      C.6   Level 3—Job-specific skills evaluation ................................................... 90
      C.7   Level 3—Training objectives .................................................................. 91
D.0         Appendix: Level 4 sample survey questions ........................................... 92
      D.1     Level 4—Business impact ...................................................................... 92


                                                                                              Page EV–5 of EV–105
                                                                                                  Evaluation—Guide
                                                                           R:HRWorkforce DevelopmentProjects and
                                                    InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement                   Version 2012.03.19



      D.2      Level 4—Test results ............................................................................. 93
      D.3      Level 4—Learning gain .......................................................................... 94
      D.4      Level 4—Skills gain ................................................................................ 95
      D.5      Level 4—Changes to business performance .......................................... 96
      D.6      Level 4—Business performance/impact measures (fixed and open) ...... 97
      D.7      Level 4—Financial impact (including ROTI) ........................................... 99
E.0       Appendix: Additional survey questions ................................................. 100
      E.1   Open question modifiers and extensions.............................................. 100
      E.2   Knowledge and skills tests (question examples) .................................. 101
F.0       Frequently asked questions (FAQs) ....................................................... 103
      F.1    How do I get a user account to create surveys in Checkbox ................ 103
G.0    Glossary ................................................................................................... 104
   G.1    Definitions ............................................................................................ 104
   G.2    Formatting convention.......................................................................... 104
   G.3    Colour coding ....................................................................................... 104
   G.4    Hyperlinks ............................................................................................ 104
   G.5    Acronyms ............................................................................................. 105




                                                                                           Page EV–6 of EV–105
                                                                                               Evaluation—Guide
                                                                        R:HRWorkforce DevelopmentProjects and
                                                 InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




1.0        Introduction
Evaluation is defined as the act of appraising to assess value.1
Evaluation identifies and promotes best practice in planning and implementation
methodology, and ensures the delivery and creation of Workforce Development
products and services is performed at the most efficient and productive level.
Effective evaluation of projects and programs ensures consistent, high quality
services and products with strong alignment to strategic goals and objectives for all
teams’ work across the Workforce Development Directorate.
Evaluation also assists in the expansion and roll-over of projects to create
sustainable programs by providing a clear and focussed relationship between
deliverables, objectives and strategic goals as targeted outcomes and responsibilities
in the management process.


Evaluation utilises descriptive foci to assist the evaluator to get into a specific frame-
of-mind and effectively evaluate the project or program from a variety of strategic
viewpoints. Project and program managers are most likely to evaluate their own
projects and programs.
Evaluative foci have been described from the Department’s Improvement and
Accountability framework (DIAf)2.
Some of the foci this guide will assist you to evaluate against include your project or
program’s:
          attention to organisational culture
          effective collation, use and reporting of data
          effective use of human, financial and physical resources
          embedding of leadership and responsibility
          focus on development and continuous improvement
          maintenance of scope and direction
          stakeholder management
          sustainability over the medium to long term
          systemic thinking and integration


The three evaluative processes described in this guide are designed to assist officers
and managers strategically prepare and evaluate projects and programs, including:
          Evaluative review of a workgroup or project team
          Strategic evaluation of a project
          Evaluation of an ongoing program, with a focus on knowledge and skills
           transfer




1
    http://dictionary.reference.com/browse/evaluation?s=t
2
    www.decd.sa.gov.au/quality/pages/quality/26420/


                                                                                       Page EV–7 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




2.0       Background
Prior to 2012 the Workforce Development directorate did not have a consistent
framework to evaluate the Directorate’s projects and programs.
Individual teams within the Directorate use a number of different and differently
applied evaluation tools, historical or mandatory evaluative processes without
scheduled or regular continuous improvement analysis of their evaluative processes,
effectiveness and outcomes.
The Department’s enterprise Registered Training Organisation (RTO), Organisation
and Professional Development Services (OPDS), use:
         national VET surveys for participants and line managers of participants
         AQTF compliance requirements
The Directorate’s Quality Leadership programs use:
      
The Directorate’s Teacher Quality projects and programs use:
      
The Directorate’s Performance and Development projects and programs use:
      
The Directorate’s Projects and Innovations projects and programs use:
      




                                                                                       Page EV–8 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




3.0      Evaluative review of a workgroup or project team
An evaluative review of a workgroup or project team is recommended once per year;
however, an evaluation may be prudent following any significant change or
development within the team; for example, new leadership, new location, change in
organisational strategic plan, or prior to the planning phase of a significant project.
The evaluation is designed as a self-review to identify and promote collaborative
discussion on opportunities for continuous improvement and increased effectiveness
that aim to improve the productivity, efficiency and capacity across the team.
Analysis of the survey responses provides specific direction to improve team
engagement, satisfaction, focus, responsiveness and resource use.
The evaluation encompasses the Department’s nine DIAf improvement principles.

DIAf corporate self-review snapshot survey
This survey is stored within the Department’s Checkbox survey tool under the
following URL < http://www.decssurveys.sa.edu.au/online/selfreveiw.aspx > and is
accessible through the internet to all Directorate officers.
The survey is password protected with the current password “opds”.
The survey details key criteria to consider as a lead into self-review of any team's
performance and operations.
Team members should complete the survey individually and encouraged to be
honest, noting all responses are anonymous3.
After all team members have been given sufficient time to comfortably respond4, the
responses should be collated to provide data for collaborative discussion.
All the survey's focus questions are answered on a ratings scale from Strongly
Disagree to Strongly Agree. This provides an opportunity within the principle analysis
to index the team’s aggregate score and accurately report on the team’s
improvement trend and recommended foci for collaborative discussion5.
There are no text or long answer questions in the survey; these types of comments
should be raised in the collaborative discussion following the survey's completion and
principle analysis6.
After the survey is closed it is important to allow and allocate adequate time for a
collaborative discussion with all team members present at a team meeting or other
dedicated meeting time.
At the collaborative discussion all members of the team are encouraged to discuss
the Focus on core business and two other focus principles identified through the
principle response analysis as the greatest opportunities to improve effectiveness
within the team.



3
  Preview and analysis of the survey responses should not begin until after a sufficient
number of responses have been received to ensure anonymity for the respondents.
4
  As the survey is accessible over the internet, some team members may feel more
comfortable completing the survey out-of-office, after hours or at home.
5
  Indices should not be compared between teams; however, all staff within the Directorate can
be anonymously surveyed at the same time to achieve a current Directorate-level index.
6
  Analysis of survey responses should be carried out by an independent and skilled analytical
or evaluative Directorate officer; for example, the Directorate’s Data Management and
Analysis Officer or Senior Consultant, Evaluation and Continuous Improvement.


                                                                                       Page EV–9 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19



The collaborative discussion should encompass:
    a) evidence used by team members to score the focus questions
    b) what the next steps would be for improvement and who will be responsible for
       managing implementation7
    c) ongoing self-review processes including evaluation of the implementation of
       the continuous improvement project
    d) the data and other evidence that would be gathered to monitor and evaluate
       improvement over time8


Improvement principles
The nine DIAf improvement principles are:
    1. Focus on core business
    2. Think systemically
    3. Share leadership
    4. Attend to culture
    5. Listen and respond
    6. Make data count
    7. Set directions
    8. Target resources
    9. Continuously improve


Improvement principle criteria
For each improvement principle there are four criteria9 that can be scored on a
ratings scale from Strongly Agree to Strongly Disagree by each individual team
member.

Focus on core business
    1.1    Team goals are clear, known and used to drive decisions
    1.2    The team has high service and delivery standards that result in high quality
           outcomes
    1.3    Team members are committed to the team’s goals
    1.4    The team’s plans, processes and practices work effectively to support team
           members to achieve goals

7
  Implementation of continuous improvement should be managed using the project
management framework laid out in the Directorate’s Information Management System (IMS)
and using the PDSA (plan, do, study, act) continuous improvement framework cycle.
8
  See the chapter in this Evaluation Guide on strategically evaluating a project.
9
  Note that principle criteria contain more than one question that may cause and result in
skewing of responses where a respondent may agree with part of the criteria and disagree
with another part. If this issue is raised, respondents should score the criteria with their lowest
desired response rating to ensure the improvement principle has the best opportunity to be
discussed for improvement by the team.


                                                                                     Page EV–10 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




Think systemically
    2.1    Political, system and contextual issues are identified and strategically
           addressed in plans and practices
    2.2    Effective research and development processes enable team members to
           improve operations, outcomes and service delivery
    2.3    Internal management processes are routinely reviewed to continuously
           improve operations
    2.4    Effective partnerships exist with key stakeholders, other business units and
           professional groups to support the achievement of unit goals

Share leadership
    3.1    Leaders provide clear direction and supportive leadership, and take an
           effective stance appropriate to the individual/situation to achieve agreed
           outcomes
    3.2    Leadership is shared with strategies and processes to build the leadership
           capacity of individuals and leadership density of the business unit
    3.3    Leaders support effective business unit management through a focus on
           professional learning for team members and themselves
    3.4    Leaders ensure change is managed positively and successfully, with
           workload balance and direction sustained

Attend to culture
    4.1    A positive workplace culture supports team members to work with
           enthusiasm, commitment, energy and the business unit to achieve success
    4.2    Team members’ roles and responsibilities are clearly known and
           professional team interactions optimise success
    4.3    Professional development and performance management processes
           provide team members with recognition, support and feedback to develop
           expertise
    4.4    Culture and morale building processes effectively support positive team
           member interactions and address issues and concerns

Listen and respond
    5.1    Quality partnerships are deliberately developed with key clients and
           stakeholders to achieve outcomes
    5.2    Communication processes provide information to and from clients and
           stakeholders to improve service delivery and outcomes
    5.3    Decision making structures are effective with high levels of team member
           and stakeholder input, support for, and engagement in decisions
    5.4    Commitment to quality service delivery and responsiveness by all team
           members provides high levels of satisfaction and positive client
           relationships




                                                                                     Page EV–11 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




Make data count
    6.1    Effective data management processes are in place to collect, store and
           access reliable data
    6.2    Multiple measures of data are analysed and used to inform improvement
           directions, evaluate programs and report on outcomes
    6.3    Data is used to identify root causes and variation to targeted improvement
           efforts while monitoring the effectiveness of implementation strategies
    6.4    Data creates knowledge and learning for the team, organisation and system
           to inform decisions on development and innovation

Set directions
    7.1    An explicitly stated vision, values and purpose developed in collaboration
           with team members drives business unit decisions, plans and directions
    7.2    Planning processes build team members’ capacity and expertise to achieve
           the vision and continuously improve outcomes
    7.3    Communication, monitoring and evaluation of planning processes occurs
           with high levels of team member involvement and ownership
    7.4    Strategic plans are integrated and enacted in daily operations to ensure
           strategic directions are achieved

Target resources
    8.1    Effective resource management systems identify, support and develop the
           team’s human, financial and physical resources
    8.2    Resources are targeted to achieve successful outcomes with processes in
           place to review resource needs and effectiveness
    8.3    Assets and resources are acquired, organised and maintained to support
           performance
    8.4    Risk management processes ensure prudent financial management,
           regulatory compliance and safe workplace practices

Continuously improve
    9.1    Effective, known improvement processes support team members to ensure
           that goals are achieved and outcomes are continuously improved
    9.2    Rigorous, regular self-review processes occur, with team members
           involvement and engagement, to monitor outcomes, evaluate progress and
           inform future directions
    9.3    A commitment to continuous improvement is evidenced by routine policy
           review and development cycles, and effective document and records
           management
    9.4    The team’s members develop integrated, sustainable and systemic
           programs, projects and products to achieve business unit goals and
           respond to the emerging needs of stakeholders




                                                                                     Page EV–12 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




Principle analysis
The principle analysis of the survey’s responses should be calculated per
improvement principle focus and include:
         an aggregate score of the responses to the criteria for each improvement
          principle (focus)
         the total number of responses and total number of possible responses (team
          members)
         a graph displaying the distribution of relative scores across the focus and its
          four criteria independently

Aggregate score
The aggregate score may:
         remove outliers—where the aggregate of one rating score is significantly
          inconsistent with the median or mode rating score
          (eg out of 25 responses: one response Strongly Disagrees while 24
          responses Agree or Strongly Agree; the Strongly Disagree response may be
          removed from the analysis)10
         be reported as a definitive modal score11—if the modal score aggregate
          exceeds the sum aggregate of all other rating scores
          (eg out of 25 responses: 16 responses are Agree while 9 responses are
          other ratings; the aggregate score for this focus can be reported as Agree)
         be reported as a definitive median score12—if the aggregate scores per rating
          appear as a balanced bell curve
          (eg out of 25 responses: 1 Strongly Disagree, 3 Disagree, 6 Neither, 9 Agree,
          6 Strongly Agree; the median score for this focus can be reported as Agree)
         be reported as a mean or average index—with each rating given an linear
          aggregate multiplier
          (eg out of the above bell curve distribution with an aggregate multiplier of x1)
Rating         Strongly Disagree        Disagree      Neither     Agree     Strongly Agree
(Multiplier)            (x1)               (x2)         (x3)       (x4)           (x5)           TOTAL
     Score               1                   3            6          9              6
Aggregate                1                   6           18         36             30               91
                                                                          Possible maximum         125
                                                                                    INDEX =        3.64
                                                                                      Agree      (72.8%)

The aggregate score for each improvement principle focus should always be
calculated and reported consistently.




10
   If there is a significant cause to the response removed from the analysis this is expected to
be raised during the collaborative discussion.
11
   The modal score is the distinct rating with the highest aggregate.
12
   The median score is the exact middle rating when all scores are accurately ordered from
lowest to highest.


                                                                                     Page EV–13 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




Graphing foci scores
The distributive graph should be chosen and produced to provide a highly visual
discussion prompt for the collaborative discussion; for example:




                                                                                     Page EV–14 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




Collaborative discussion
The collaborative discussion is the most important part of the evaluation.
The discussion enables team members to focus on specific areas for improvement,
identify barriers and work as a team to trouble-shoot solutions to issues that impact
the self-identified effectiveness of the team.
Only three foci should be discussed in this meeting including the Focus on core
business plus two other foci recommended by the independent principle analyst as
areas which have a very low aggregate score or where the aggregate score does not
appear representative of a clear majority; (eg out of 25 responses: 12 Strongly
Disagree, 12 Agree, 1 Strongly Agree).
The collaborative discussion should occur as a dedicated meeting with no other
agenda items, in a closed meeting room set up for brainstorming (whiteboards,
butcher’s paper, post-it notes), and all team members present. If possible the
discussion should be led by an independent facilitator.
The meeting must allow for anonymous reporting of causative factors leading to the
opportunity for improvement; (eg all team members may be asked to write
anonymously a possible cause for the low criteria or focus score on a post-it note and
pass those up to the independent facilitator who will ensure no individuals are
identified while the causes are being discussed).
The facilitator should use techniques like the 5-Why’s13 to determine root causes of
the score and to provoke collaborative discussion on potential resolutions.
After resolutions have been brainstormed, the facilitator should lead the team in
identifying the resolution project to be recommended
The project proposal should be development through project management
framework, with the team brainstorming components including:
         potential risks
         implementation schedule
         stakeholders to be engaged
         project team and responsibilities
         objectives and deliverables to be achieved
         evaluation process to ensure completion
         and measurable, reportable achievement14
The improvement project manager or team should report the improvement project’s
ongoing status back to the whole team as an agenda item at every future normal
team meeting until the project is closed.




13
   To identify the root cause of an issue, ask Why the issue occurred, then ask Why the cause
of the issue occurred, then ask Why the cause of the cause of the issue occurred, to a total of
5-Why’s.
14
   See the chapter in this Evaluation Guide on strategically evaluating a project.


                                                                                     Page EV–15 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




3.A      Appendix: Example—DIAf corporate self review survey




                                                                                     Page EV–16 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                     Page EV–17 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                     Page EV–18 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                     Page EV–19 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                     Page EV–20 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                     Page EV–21 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                     Page EV–22 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                                                                                              Page EV–23 of EV–105
                                                                                                                                                                   Evaluation—Guide
                                                                             R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                                                                                              Page EV–24 of EV–105
                                                                                                                                                                   Evaluation—Guide
                                                                             R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                                                                                              Page EV–25 of EV–105
                                                                                                                                                                   Evaluation—Guide
                                                                             R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                                                                                              Page EV–26 of EV–105
                                                                                                                                                                   Evaluation—Guide
                                                                             R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                                                                                              Page EV–27 of EV–105
                                                                                                                                                                   Evaluation—Guide
                                                                             R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                                                                                              Page EV–28 of EV–105
                                                                                                                                                                   Evaluation—Guide
                                                                             R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                                                                                              Page EV–29 of EV–105
                                                                                                                                                                   Evaluation—Guide
                                                                             R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                                                                                              Page EV–30 of EV–105
                                                                                                                                                                   Evaluation—Guide
                                                                             R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                                                                                              Page EV–31 of EV–105
                                                                                                                                                                   Evaluation—Guide
                                                                             R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




4.0        Strategic evaluation of a project
A project is a temporary measure with distinct start and end dates comprising a time
constraint to meet specific goals and objectives, and produce deliverables within a
defined budget and limited resources, in order to achieve beneficial, value-adding
change.15
A program does not have a defined end date and usually contains multiple repetitious
project instances defined for example, per annum.16
Projects and programs should be evaluated by the project or program manager, or a
specified third party, during the:
          project concept or proposal phase
          throughout the project implementation as required
          post-project review or closure phase


Strategic evaluation of a project or program enables evaluation against its:
          Objectives
          Deliverables
          Strategic alignment
          Evaluation measures
Project objectives align to strategic plans and goals, and are achieved and evidenced
through the production of project deliverables throughout the implementation of the
project.
The project’s deliverables are evaluated to ensure they have effectively achieved the
project’s objectives and strategic alignment through evidence of achievement.
This evaluation framework assist the project or program manager evaluate their
project’s deliverables, objectives, strategic alignment and evaluative processes
against foci developed from the Department’s Improvement and Accountability
framework (DIAf).


All Workforce Development projects and programs including courses and events
should be recorded in the Directorate’s information management system (IMS).
< decsgla01user3GroupsHRWorkforce DevelopmentIMSinterfacesProject
Management.mdb >
This evaluation framework is planned to be integrated into the IMS Work Program
and Project Management interface.




15
     For example, the 2012 South Australian Public Teaching Awards is a project.
16
     For example, the South Australian Public Teaching Awards is a program.


                                                                                     Page EV–32 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




Objectives
Objectives to be achieved through the project should be recorded into the IMS.
An objective is a key benefit achievable by the project that aims to meet a strategic
direction, plan or vision for the Department.
Objectives are usually described in the format: To..., in a way that..., so that...
Objectives have no physical substance; they are not created in a visible way.17
Score             Description                                     Example
Undeveloped       objectives resemble a brain-storming            To improve image
                  list, are vague and have no indication
                  of change or outcomes
Developing        objectives are stated broadly and infer         To improve the image of the
                  or imply non-specific outcomes or               Department
                  changes in practice
Functioning       objectives are clear about the                  To improve the image teachers have
                  outcomes to be targeted or improved             of the Department recognising them
                  but the change is stated in broad non-          as part of the workforce
                  specific terms
Strategic         objectives are linked to a strategic            To improve the image teachers have
                  objective in a plan and expressed in            of the Department recognising them
                  terms of the outcomes to be targeted            as an important part of the education
                  or improved                                     and care workforce
Embedded          objectives are STRATEGIC, based on              To improve the image teachers have
                  best practice, aligned to increase              of the Department recognising them
                  effectiveness and consistency, and              as the part of the education
                  embedded with a specific objective in           workforce that provides excellence
                  a strategic plan                                in education and care

Please update the objectives of your project in the IMS before scoring.




17
     Compare with deliverables.


                                                                                     Page EV–33 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




Deliverables
Deliverables to be created during the project’s implementation should be recorded in
the IMS.
A deliverable is a key product that will be created by the project providing physical
evidence of the project's activity.18
Deliverables include key performance indicators (KPIs) as statistics and are used as
proof the project is meeting its Objectives.
Score             Description                            Example
Undeveloped       are when no deliverables are           None
                  stated and implementation
                  does not produce any physical
                  evidence
Developing        deliverables are common or             Hold an awards ceremony for 2012
                  historically repeated with no
                  significant improvement
                  evident
Functioning       deliverables include data              Hold a ceremony for 100 award winners in
                  related to the project but             2012
                  lacking specificity against
                  changes or improvements to
                  be achieved
Strategic         deliverables are assigned to           The Workforce Recognition Officer will event
                  specific team members, are             manage the 2012 SA Public Teaching
                  demonstrable including and             Awards ceremony for up to 100 teachers
                  supported by data, and                 recognised and nominated in the last 12
                  expressed as SMART                     months for outstanding teaching practice
                  (specific, measurable,
                  achievable, relevant, timely)
Embedded          deliverables are STRATEGIC,            The Workforce Recognition Officer will event
                  and include agreements for             manage the 2012 SA Public Teaching
                  continuous support and                 Awards ceremony promoting excellence in
                  improvement with deliverables          education and care for up to 100 teachers
                  for post-project analysis and          recognised and nominated in the last 12
                  review                                 months for outstanding teaching practice
                                                         using a documented up-to-date event
                                                         management process including a survey of
                                                         participants on the event's delivery

Please update the planned and achieved deliverables of your project in the IMS before
scoring.




18
     Compare with objectives.


                                                                                     Page EV–34 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




Strategic plans and alignment
A link from each project objective to a specific strategic plan’s objective or goal
should be recorded against each project objective in the IMS.
Linking objectives to a strategic direction, plan or vision is an important analysis of
the value of your project.
Score             Description                                                                    Example
Undeveloped       There are a high number of competing strategies which are very
                  broad, are management or compliance requirements, or are
                  determined by the Department
Developing        There are a range of broad strategies areas, based on complex
                  targets that are a mix of management and improvement issues,
                  already described by the Department
Functioning       There are a number of strategies that are mainly improvements or
                  developments, identified from strategic planning, Departmental or
                  national objectives
Strategic         There are a number of strategies that clearly define continuous
                  improvement integrated with Departmental or national objectives,
                  determined through data analysis, agreed to by staff and
                  described against a particular field of expertise
Embedded          There are a number of strategies that all staff agree from data
                  analysis and stakeholder consultation, that are key objectives to
                  focus for continuous improvement, that expressed against
                  particular fields of expertise will concurrently and strategically
                  achieve Departmental or national objectives

Please update your project's objective links to strategic plans in the IMS before
scoring.




                                                                                     Page EV–35 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement             Version 2012.03.19




Evaluation measures
Projects managed through the IMS have measures enabling evaluation to be
described and resulted through your project's objectives, deliverables and strategic
links.
Evaluation includes your project's ongoing status updates as well as the final project
review closure report.
Score             Description                                                     Example
Undeveloped       Data may be collated but is not used to monitor                 # participants
                  progress, support status reporting or evaluate the
                  effectiveness of the project
Developing        Measure of progress are merely numbers or                       gender distribution of #
                  deliverables ticked as completed or achieved with               participants
                  minimal or no evaluation of the relative change or              survey of participants
                  improvement achieved                                            interest
Functioning       Data is used throughout the project to monitor                  rate of registration of
                  and report on the progress and achievement of                   interest
                  targets, deliverables and objectives against
                  strategic plans
Strategic         Specific data is regularly collated and used to                 regional and index of
                  monitor the progress of the continuous                          disadvantage
                  improvement changes throughout the project's                    distributions of rate of
                  lifecycle, with regular review of the completion of             registration and
                  deliverables to support the achievement of                      achievement vs student
                  objectives against strategic plans by all project               outcomes (NAPLAN)
                  team members
Embedded          Evaluation is STRATEGIC with the analysis of                    post event survey and
                  multiple measures of improvement and the                        data retrieval
                  regular collaboration and input from all project                quantitatively analysing
                  team members to evaluate the project's progress                 changes and
                  and refine and redefine the objectives and next                 improvement in
                  steps towards the most effective deliverables to                education and care
                  achieve the strategic outcomes of the project                   activity

Please start your post project review in the IMS before scoring, including:
 • what worked
 • what didn't work
 • lessons learned
 • your recommendations for future project managers




                                                                                     Page EV–36 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




Focus areas
The following focus areas may be used to categorise the project’s objectives,
deliverables, strategic alignment and evaluation measures for projects where the
implementation of the evidentiary deliverables to achieve the project objectives
aligned with strategic plans is significantly large or complex.

Focusing on development or improvement
The project produced evidence of continuous improvement in achievement/outcomes
for stakeholders, with challenging targets for ongoing improvement and quality
practice, while creating an ethos with high expectations and consistent understanding
to drive, develop and improve policy, practice and performance.

Designed and implemented with systemic thinking
The project is integrated with wider systems and is committed to improve these
systems to create an aligned and effective integrated system that supports the
Department's continual improvement with targets for achievement appropriate to the
context of community needs and aspirations.

Enables individual leadership
The project develops leadership within the project's team members and stakeholders
enabling them to exhibit principled and visible responsibility that is shared and
fostered throughout the project through effective project management expertise and
capacity at all levels to achieve the project's objectives, deliverables and strategic
goals.

Attentive to organisational culture while transforming capacity
The project intentionally creates a culture that involves individuals and groups in
transforming the capacity of the system through a positive culture with high levels of
team and stakeholder satisfaction, morale and support for individuals to grow and
improve their performance.

Managed stakeholders through effective listening and responsiveness
The project enables team members and stakeholders to have an active voice and be
responsive to current and future needs of the Department using a client-focussed
approach to communication, risk management, prioritisation, and organisation of the
project's components and responsibilities.

Effectively collated, used and reported data
The project creates or gathers the necessary information and knowledge required
from data sources including stakeholder needs to strategically evaluate and improve
outcomes through an informed, structured and organised approach with a clear
evidence base for decision-making and recommendations as well as identifying clear
drivers for continuous improvement opportunities.

Maintained direction and scope throughout the project
The project planning process, documentation and stakeholder communication
explicitly states values, vision and purpose within its objectives, deliverables and
strategic links, and all documentation is consistently up-to-date and available to all
project team members.




                                                                                     Page EV–37 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19



Innovatively and effectively aligned resources
The project innovatively targets resources to most effectively align all physical,
human and financial resources with project needs through sustainable resource
management to achieve successful and efficient outcomes.

Building a sustainable future including continuous improvement
The project achieves improvement for the Department through a structured
continuous improvement cycle with regular evaluation and review of existing related
processes and activity.


Scoring
Scoring of your project’s objectives, deliverables, alignment to strategic plans, and
evaluation measures is recorded as a single rating across each component either as
a whole, or categorised into focus areas.
     1.        Undeveloped
     2.        Developing
     3.        Functioning
     4.        Strategic
     5.        Embedded

Undeveloped
An undeveloped rating refers to objectives, deliverables, strategic alignment and
evaluation measures that are undocumented, unrecorded and unable to be reviewed
by an appropriate third party.

Developing
A developing rating refers to objectives, deliverables, strategic alignment and
evaluation measures that are drafted, or only documented on the initial project plan,
concept or proposal.

Functioning
A functioning rating refers to objectives, deliverables, strategic alignment and
evaluation measures that have been or are intended to be ticked off as achieved or
not achieved; however, the project’s objectives or deliverables are not linked, related
or aligned to strategic plans, goals or objectives.

Strategic
A strategic rating refers to objectives, deliverables, strategic alignment and
evaluation measures where the project’s objectives and deliverables are linked,
related, aligned and described against strategic plans, goals or objectives.

Embedded
An embedded rating refers to objectives, deliverables, strategic alignment and
evaluation measures where the project only includes objectives that are linked to
strategic objectives; with all project objectives evidenced by deliverables (with no
non-strategic, orphan objectives or deliverables); and with the project plan,
implementation, delivery and closure evaluation focused on strategic achievement.


                                                                                     Page EV–38 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




4.A      Appendix: Example—strategic project evaluation




                                                                                     Page EV–39 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                     Page EV–40 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                     Page EV–41 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                     Page EV–42 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                     Page EV–43 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                     Page EV–44 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                     Page EV–45 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                     Page EV–46 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




                                                                                     Page EV–47 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




5.0        Evaluation of an ongoing program, with a focus on
           knowledge and skills transfer
This evaluation is designed to encompass both effective training (improvement of
programs) and training effectiveness (implementation of skills to improve
organisational effectiveness).
The Kirkpatrick19 model will be used as the structural framework in the evaluation of
Workforce Development training programs.
This model has the following four levels.
      Level 1—evaluation of feedback from participants and stakeholders
      Level 2—evaluation of physical content, resources and deliverables
      Level 3—evaluation of application and change to organisational processes
      Level 4—evaluation of strategic outcomes and measurable benefits
Levels 1–2 provide instructive feedback to program deliverers to update and improve
their relationship with participants, and to project managers in the roll-over of project
initiatives to ongoing programs for the future program manager.
Levels 3–4 produce summation of strategic value and effectiveness in the
implementation of a project and ongoing program delivery reportable to senior
management.
Some methods of evaluation include:
          asking for self-evaluation from participants and facilitators
          testing participants’ knowledge, understanding and decision-making skill
          observing participants’ performance
          examining organisational business results
          comparing social media learning with traditional learning intervention
          seeing how much productivity is lost or gained from time required
Results from evaluations can be analysed in global categories to calculate the
effectiveness of a type of training for the organisation; for example, all leadership
program evaluations.


Sample evaluation survey questions are listed in Appendices A–E.




19
     www.kirkpatrickpartners.com


                                                                                     Page EV–48 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




Level 1—evaluation of motivation and reaction
The first level of evaluation is used to determine how well your participants engage
with the registration and learning process.
It is used to identify why participants have enrolled in the program, what they
perceive are needs in their workplace that the program will have a positive impact on
to improve their working environment, and what their learning expectations are from
participation in the program.
The motivation and reaction evaluation informs on how appealing, relevant and
effective the learning content and delivery is to the individual participants, measuring
how well the learning engagement processes and program reputation work.
Even if all the other levels are effectively covered by the program, missing level 1 can cause
participants to fail to see a purpose in the learning and they may disengage from the program.
Effective evaluation of level 1 motivation and reaction to the program to measure the
engagement of participants to its structure and marketing can be collated and
analysed through:
         reaction sheets
         surveys
         focus groups
         interviews
Participants may self-assess their impression of the program pre-, during and post-
participation against any improvement rating scale with the following foci:
         relevance
         specific
         practical
         accessible
         social




                                                                                     Page EV–49 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




Level 2—evaluation of learning content
The second level of evaluation is used to improve the quality and effectiveness of the
learning content, structure and delivery.
It is used to update, refine and ensure best practice in the knowledge, skills, and
resources provided to and developed by participants who participate in the
facilitator’s program.
The learning content evaluation informs on what participants learned and the extent
to which knowledge and skills were gained by the participants, including the degree
of effectiveness that the learning delivery achieved to transfer the content, knowledge
and skills to the participants.
Even if all the other levels are effectively covered by the program, missing level 2 can cause
the participants to devalue the learning and consider the program to be out-of-date and
irrelevant.
Effective evaluation of level 2 learning content in the program to measure successful
content design and development, delivery methods and achievement can be collated
and analysed through:
         pre and post training results
         written knowledge tests
         role-play and simulation
         activities and games




                                                                                     Page EV–50 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




Level 3—evaluation of performance and behaviour
The third level of evaluation is used to report on changes and increases in capacity of
job performance resulting from participation in the program, driving accountability,
measuring effectiveness and value to the organisation, and enabling appropriate
resource and support allocation.
It is used to prove value of participation in the program through identifying changes in
the way employees behave and perform in their working environment to increase the
efficiency, productivity and quality of their work using newly learned skills and
knowledge.
The performance and behaviour evaluation informs of the capability of participants to
effectively uptake and perform newly developed skills, and the degree that learned
skills and knowledge actually transfer to and are used in the working environment,
post-participation.
Even if all the other levels are effectively covered by the program, missing level 3 can cause
the program to be devalued by staff and management as they consider the skills and
knowledge learned through participation to be academic and ineffective in the real workplace.
Effective evaluation of level 3 performance and behaviour following participation in
the program to measure successful transfer and implementation of learning from the
training to working environments, can be collated and analysed through:
         surveys
         interviews
         focus groups
         observations
         work reviews
Participants may self-assess changes in their working environment following
participation in the program against any improvement rating scale with the following
foci:
         performance
         confidence
         contribution
         engagement
         retention
         customer satisfaction
         business success
         systems use
         drivers and strategic alignment




                                                                                     Page EV–51 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




Level 4—evaluation of impact and results
The fourth level of evaluation is used to report on the tangible organisational
outcomes achieved through the implementation of skills developed by participation in
the program.
It is used to graphically demonstrate the productive impact, outcomes and results of
participation showing organisational improvement.
The impact and results evaluation provides quantitative results and scoring of added
organisational value including reduced costs, improved quality, increased production
and efficiency indices, and enables calculation of return on investment (ROI) from the
program.
Even if all the other levels are effectively covered by the program, missing level 4 can cause
the program to have reduced priority and acknowledgement by C-level and Executive
management as they may not easily identify a positive impact on the organisation’s bottom
line by supporting the program.
Effective evaluation of level 4 impact and results to measure the outcomes of
implementation of learning to achieve described targets can be collated and analysed
through:
          borrowed metrics from other data systems including human resources data
          surveys
          focus groups


Decision-makers prefer the results of level 4 evaluations, although not necessarily in
dollars and cents. For example; a study of financial and information technology
executives found that they consider both hard and soft returns when it comes to
customer-centric technologies, but give more weight to non-financial metrics (soft),
such as customer satisfaction and loyalty.20
These evaluation results enable real business results to be connected to training
programs, including:
          operational efficiency
          compliance
          retention of top talent
          customer satisfaction
          sales volume
These results also identify through analysis, the gaps and needs of the organisation
that can be filled through training and skills development.




20
     Hayes 2003


                                                                                     Page EV–52 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19



This level of program evaluation may be met with a more balanced approach or
scorecard21 from four perspectives:
          Financial—a measurement that shows a monetary return or impact such as
           how the output from a process is improved (financial can be either soft or
           hard results)
          Customer—improving an area in which the organisation differentiates itself
           from competitors to attract, retain, and deepen relationships with its targeted
           customers
          Internal—achieving excellence by improving processes as supply-chain
           management, production or support process
          Innovation and learning—ensuring learning packages support a climate for
           organisational change, innovation, and the growth of individuals


Evaluative results from the above scorecard are preferred by management but may
be supplemented with the more commonly provided levels 1–2 quantitative results
listed below.
          How many people (participants) will receive the training?
          How often the training both can and is planned to be repeated; noting that
           online training courses can be developed once and re-used with relatively
           low ongoing costs, whereas the volume of financial, human and physical
           resources required for classroom delivery continues to increase
          How many total hours of learning have been successfully achieved?
          A comparison of the cost of initial development (initial or single class), with
           the reducing cost to maintain and deliver multiple instances of the same
           content
          Overall resource costs involved in preparing, running and evaluating
           programs, in both dollars ($) and hours.
          Do participants and/or facilitators need to travel, be accommodated, and
           supplemented at the workplace while they are training?
          What proportion of programs were evaluated, and the distribution of
           stringency of those evaluations?
          What proportion of programs were evaluated by the program manager,
           versus those evaluated by an independent third party evaluator?
          The effective distribution of evaluation results.
          How many programs or instances were planned using or referencing
           evaluation results from previous programs or relevant projects, before
           implementation?
          What evidence is there of the implementation of evaluation recommendations
           in the delivery of programs?




21
     Kaplan, Norton 2001


                                                                                     Page EV–53 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




A.0       Appendix: Level 1 sample survey questions
The first level of evaluation is used to determine how well your participants engage
with the registration and learning process.22

A.1       Level 1—About the training needs analysis
1.        This training needs analysis asks about [DELETE AS APPROPRIATE] the
          business need for training / your individual learning needs and preferences.
2.        The assessment is made up of [ADD NUMBER] questions and should take
          about [ADD NUMBER] minutes to complete. Please answer the questions as
          fully as possible.
3.        Please complete the assessment by [ADD DETAILS].
4.        What will happen with your responses?
             [DELETE AS APPROPRIATE]
             Your responses will be used to help decide on the most appropriate
              training needed.
             Your responses will be anonymous and feedback will be reported for all
              respondents in generalised form only.
             All responses will be analysed and reported to [ADD DETAILS]. Findings
              are due to be reported in [ADD DETAILS].
             A copy of the assessment report will be available from [ADD
              LOCATION/PARTICIPANT] on [ADD DATE].
5.        Any questions?
6.        If you have any questions about this assessment, please contact [ADD
          DETAILS].




22
     The following questions have been adapted from < www.trainingcheck.com >.


                                                                                     Page EV–54 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




A.2      Level 1—About the training
1.       What was the name of the training/activity you attended?
2.       Which training event did you attend?
3.       Which sessions did you attend? Choose as many as apply.
             All sessions
             Session 1
             Session 2
             Session 3
4.       Which modules did you attend?
             All
             Module 1 only
             Module 2 only
             Other, please specify
5.       How many sessions did you attend in total?
6.       Where was the training/activity held?
7.       Which location did you attend?
8.       What was the date of the training?
             dd/mm/yyyy
9.       Which date did you attend?
10.      Which times did you attend?
11.      Which level did you attend?
12.      What type of training/activity was it?
             Classroom-based
             Web-based/e-learning
             On-the-job learning
             Coaching/mentoring
             Project work
             Job shadowing
             Other, please specify
13.      What was/were the name(s) of the trainer(s)?
14.      Who was the trainer?




                                                                                     Page EV–55 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19




A.3      Level 1—About the participant
1.       Your name:
2.       Your organisation:
3.       Your job title/grade:
4.       What is your job title/grade?
             [FIRST JOB TITLE / GRADE / CLASSIFICATION]
             [SECOND JOB TITLE / GRADE / CLASSIFICATION]
             [THIRD JOB TITLE / GRADE / CLASSIFICATION]
             Other, please specify
5.       Which of these best describes your job role?
             Senior manager
             Manager
             Supervisor
             Employee
             Other, please specify
6.       On what basis are you employed?
             Full-time
             Part-time
             Contract/agency
             Other, please specify
7.       Which department/section do you work in?
8.       What is your manager's name?
9.       Which shift pattern do you work in:
10.      How long (in months) have you been in your current role?
11.      How long have you worked for the organisation?
             Less than one year
             1-2 Years
             3-4 Years
             5-10 Years
             11-15 Years
             16-20 Years
             21-25 Years
             More than 25 years




                                                                                     Page EV–56 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19



12.      Your age:
             Under 18
             18-25
             26-35
             36-45
             46-55
             56+
13.      What is your date of birth?
             dd/mm/yyyy
14.      Your gender:
             Male
             Female
15.      Is English your first language?
             Yes
             No
16.      What is your primary language?
             English
             [LANGUAGE 2]
             [LANGUAGE 3]
             [LANGUAGE 4]
             Other, please specify
17.      What is the highest level of education that you have completed?
             Primary/elementary school
             Secondary/high/vocational school
             Further tertiary education/vocational college
             University
             Other, please specify
18.      What is the highest grade or year of school you completed?
             Never attended school or only attended kindergarten
             Grades 1 through 7 (Primary)
             Grades 8 through 11 (High school)
             Grade 12 (High school graduate)
             Tertiary 1–3 years (TAFE or University)
             Tertiary 4+ years (Post-graduate)
             Post-tertiary (Post-graduate Honours, Masters, Doctorate)
             Other, please specify


                                                                                     Page EV–57 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement          Version 2012.03.19



19.      What is your highest qualification?
             Doctorate
             Master’s
             Honours
             Bachelor's
             Diploma
             Certificate
             None
             Other, please specify




                                                                                     Page EV–58 of EV–105
                                                                                           Evaluation—Guide
                                                                    R:HRWorkforce DevelopmentProjects and
                                             InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide
Evaluation Guide

Mais conteúdo relacionado

Semelhante a Evaluation Guide

ELDP Capital Planning White Paper_20130130
ELDP Capital Planning White Paper_20130130ELDP Capital Planning White Paper_20130130
ELDP Capital Planning White Paper_20130130Carlos Rivero
 
Mastering Project Management
Mastering Project ManagementMastering Project Management
Mastering Project Managementddonahoo
 
Organization assessment guide
Organization assessment guideOrganization assessment guide
Organization assessment guiderajatoba1
 
Project Management Project Assignment - Project Schedule & Resource Allocatio...
Project Management Project Assignment - Project Schedule & Resource Allocatio...Project Management Project Assignment - Project Schedule & Resource Allocatio...
Project Management Project Assignment - Project Schedule & Resource Allocatio...Shuchi Joshi
 
McGill Project Team Proposal for Bombardier Aviation_20150226
McGill Project Team Proposal for Bombardier Aviation_20150226McGill Project Team Proposal for Bombardier Aviation_20150226
McGill Project Team Proposal for Bombardier Aviation_20150226Leandro Eduardo Cantelli
 
Practical Implementation Tips For Implementing a Financial Planning - QueBIT ...
Practical Implementation Tips For Implementing a Financial Planning - QueBIT ...Practical Implementation Tips For Implementing a Financial Planning - QueBIT ...
Practical Implementation Tips For Implementing a Financial Planning - QueBIT ...QueBIT Consulting
 
Insights and Trends: Current Portfolio, Programme, and Project Management ...
Insights and Trends:  Current Portfolio,  Programme, and Project  Management ...Insights and Trends:  Current Portfolio,  Programme, and Project  Management ...
Insights and Trends: Current Portfolio, Programme, and Project Management ...CollectiveKnowledge
 
Upgrade To Windows 7 Soe
Upgrade To Windows 7 SoeUpgrade To Windows 7 Soe
Upgrade To Windows 7 SoeDainSanye
 
Internship project proposal report
Internship project proposal reportInternship project proposal report
Internship project proposal reportJAI KISHAN CHAURASIA
 
Rethinking learning for a volatile and uncertain future
Rethinking learning for a volatile and uncertain futureRethinking learning for a volatile and uncertain future
Rethinking learning for a volatile and uncertain futurelearnd
 
Edu 653 power point h. aldrich
Edu 653 power point h. aldrichEdu 653 power point h. aldrich
Edu 653 power point h. aldrichrich1hm
 
U.S. Department of EducationA Planning Guide for Aligning .docx
U.S. Department of EducationA Planning Guide for Aligning .docxU.S. Department of EducationA Planning Guide for Aligning .docx
U.S. Department of EducationA Planning Guide for Aligning .docxouldparis
 
Schaffner edited2 STRATEGIC PLAN AND PRESENTATION FINAL
Schaffner edited2 STRATEGIC PLAN AND PRESENTATION FINALSchaffner edited2 STRATEGIC PLAN AND PRESENTATION FINAL
Schaffner edited2 STRATEGIC PLAN AND PRESENTATION FINALStacey Schaffner-Moy
 
Deloitte Advisory RISE Case Competition
Deloitte Advisory RISE Case CompetitionDeloitte Advisory RISE Case Competition
Deloitte Advisory RISE Case CompetitionMiles Wood
 
Agile Business Intelligence - course notes
Agile Business Intelligence - course notesAgile Business Intelligence - course notes
Agile Business Intelligence - course notesEvan Leybourn
 
Curriculum Vitae of Jimmy Majumder
Curriculum Vitae of Jimmy Majumder Curriculum Vitae of Jimmy Majumder
Curriculum Vitae of Jimmy Majumder jimmy majumder
 
The North Eastern Mountain Bike Development Project
The North Eastern Mountain Bike Development ProjectThe North Eastern Mountain Bike Development Project
The North Eastern Mountain Bike Development ProjectKerry Lewis
 

Semelhante a Evaluation Guide (20)

ELDP Capital Planning White Paper_20130130
ELDP Capital Planning White Paper_20130130ELDP Capital Planning White Paper_20130130
ELDP Capital Planning White Paper_20130130
 
Bringing project learning to forefront
Bringing project learning to forefrontBringing project learning to forefront
Bringing project learning to forefront
 
Bringing project learning to forefront
Bringing project learning to forefrontBringing project learning to forefront
Bringing project learning to forefront
 
Mastering Project Management
Mastering Project ManagementMastering Project Management
Mastering Project Management
 
Organization assessment guide
Organization assessment guideOrganization assessment guide
Organization assessment guide
 
Project Management Project Assignment - Project Schedule & Resource Allocatio...
Project Management Project Assignment - Project Schedule & Resource Allocatio...Project Management Project Assignment - Project Schedule & Resource Allocatio...
Project Management Project Assignment - Project Schedule & Resource Allocatio...
 
McGill Project Team Proposal for Bombardier Aviation_20150226
McGill Project Team Proposal for Bombardier Aviation_20150226McGill Project Team Proposal for Bombardier Aviation_20150226
McGill Project Team Proposal for Bombardier Aviation_20150226
 
Project Report
Project ReportProject Report
Project Report
 
Practical Implementation Tips For Implementing a Financial Planning - QueBIT ...
Practical Implementation Tips For Implementing a Financial Planning - QueBIT ...Practical Implementation Tips For Implementing a Financial Planning - QueBIT ...
Practical Implementation Tips For Implementing a Financial Planning - QueBIT ...
 
Insights and Trends: Current Portfolio, Programme, and Project Management ...
Insights and Trends:  Current Portfolio,  Programme, and Project  Management ...Insights and Trends:  Current Portfolio,  Programme, and Project  Management ...
Insights and Trends: Current Portfolio, Programme, and Project Management ...
 
Upgrade To Windows 7 Soe
Upgrade To Windows 7 SoeUpgrade To Windows 7 Soe
Upgrade To Windows 7 Soe
 
Internship project proposal report
Internship project proposal reportInternship project proposal report
Internship project proposal report
 
Rethinking learning for a volatile and uncertain future
Rethinking learning for a volatile and uncertain futureRethinking learning for a volatile and uncertain future
Rethinking learning for a volatile and uncertain future
 
Edu 653 power point h. aldrich
Edu 653 power point h. aldrichEdu 653 power point h. aldrich
Edu 653 power point h. aldrich
 
U.S. Department of EducationA Planning Guide for Aligning .docx
U.S. Department of EducationA Planning Guide for Aligning .docxU.S. Department of EducationA Planning Guide for Aligning .docx
U.S. Department of EducationA Planning Guide for Aligning .docx
 
Schaffner edited2 STRATEGIC PLAN AND PRESENTATION FINAL
Schaffner edited2 STRATEGIC PLAN AND PRESENTATION FINALSchaffner edited2 STRATEGIC PLAN AND PRESENTATION FINAL
Schaffner edited2 STRATEGIC PLAN AND PRESENTATION FINAL
 
Deloitte Advisory RISE Case Competition
Deloitte Advisory RISE Case CompetitionDeloitte Advisory RISE Case Competition
Deloitte Advisory RISE Case Competition
 
Agile Business Intelligence - course notes
Agile Business Intelligence - course notesAgile Business Intelligence - course notes
Agile Business Intelligence - course notes
 
Curriculum Vitae of Jimmy Majumder
Curriculum Vitae of Jimmy Majumder Curriculum Vitae of Jimmy Majumder
Curriculum Vitae of Jimmy Majumder
 
The North Eastern Mountain Bike Development Project
The North Eastern Mountain Bike Development ProjectThe North Eastern Mountain Bike Development Project
The North Eastern Mountain Bike Development Project
 

Mais de DainSanye

Dain Sanye - Agile Project Management Practitioner certification
Dain Sanye - Agile Project Management Practitioner certificationDain Sanye - Agile Project Management Practitioner certification
Dain Sanye - Agile Project Management Practitioner certificationDainSanye
 
Dain Sanye press release
Dain Sanye press releaseDain Sanye press release
Dain Sanye press releaseDainSanye
 
PRINCE2 Practitioner
PRINCE2 PractitionerPRINCE2 Practitioner
PRINCE2 PractitionerDainSanye
 
LMS Facilitators Guide_preview
LMS Facilitators Guide_previewLMS Facilitators Guide_preview
LMS Facilitators Guide_previewDainSanye
 
Learning management system design specification_preview
Learning management system design specification_previewLearning management system design specification_preview
Learning management system design specification_previewDainSanye
 
LMS project synopsis 6sigma
LMS project synopsis 6sigmaLMS project synopsis 6sigma
LMS project synopsis 6sigmaDainSanye
 
ITIL certification
ITIL certificationITIL certification
ITIL certificationDainSanye
 
Windows 7 evaluation survey
Windows 7 evaluation surveyWindows 7 evaluation survey
Windows 7 evaluation surveyDainSanye
 
ICA40305 Certificate IV in Information Technology
ICA40305 Certificate IV in Information TechnologyICA40305 Certificate IV in Information Technology
ICA40305 Certificate IV in Information TechnologyDainSanye
 
PSP30104 Certificate III in Government
PSP30104 Certificate III in GovernmentPSP30104 Certificate III in Government
PSP30104 Certificate III in GovernmentDainSanye
 
Reference letter
Reference letterReference letter
Reference letterDainSanye
 
Reference letter
Reference letterReference letter
Reference letterDainSanye
 
eSee Talking With Family (instructions for using a web-camera by older people...
eSee Talking With Family (instructions for using a web-camera by older people...eSee Talking With Family (instructions for using a web-camera by older people...
eSee Talking With Family (instructions for using a web-camera by older people...DainSanye
 
Reference letter
Reference letterReference letter
Reference letterDainSanye
 
DIAf evaluation survey
DIAf evaluation surveyDIAf evaluation survey
DIAf evaluation surveyDainSanye
 
ICT Central Knowledge Base documentation
ICT Central Knowledge Base documentationICT Central Knowledge Base documentation
ICT Central Knowledge Base documentationDainSanye
 
How to use the HEAT helpdesk call-logging software
How to use the HEAT helpdesk call-logging softwareHow to use the HEAT helpdesk call-logging software
How to use the HEAT helpdesk call-logging softwareDainSanye
 
Information Management System presentation, Office For The Ageing
Information Management System presentation, Office For The AgeingInformation Management System presentation, Office For The Ageing
Information Management System presentation, Office For The AgeingDainSanye
 
Comparative analysis for J&amp;PS ICT first level helpdesk
Comparative analysis for J&amp;PS ICT first level helpdeskComparative analysis for J&amp;PS ICT first level helpdesk
Comparative analysis for J&amp;PS ICT first level helpdeskDainSanye
 

Mais de DainSanye (20)

Dain Sanye - Agile Project Management Practitioner certification
Dain Sanye - Agile Project Management Practitioner certificationDain Sanye - Agile Project Management Practitioner certification
Dain Sanye - Agile Project Management Practitioner certification
 
Dain Sanye press release
Dain Sanye press releaseDain Sanye press release
Dain Sanye press release
 
PRINCE2 Practitioner
PRINCE2 PractitionerPRINCE2 Practitioner
PRINCE2 Practitioner
 
LMS Facilitators Guide_preview
LMS Facilitators Guide_previewLMS Facilitators Guide_preview
LMS Facilitators Guide_preview
 
Learning management system design specification_preview
Learning management system design specification_previewLearning management system design specification_preview
Learning management system design specification_preview
 
LMS project synopsis 6sigma
LMS project synopsis 6sigmaLMS project synopsis 6sigma
LMS project synopsis 6sigma
 
ITIL certification
ITIL certificationITIL certification
ITIL certification
 
Windows 7 evaluation survey
Windows 7 evaluation surveyWindows 7 evaluation survey
Windows 7 evaluation survey
 
ICA40305 Certificate IV in Information Technology
ICA40305 Certificate IV in Information TechnologyICA40305 Certificate IV in Information Technology
ICA40305 Certificate IV in Information Technology
 
PSP30104 Certificate III in Government
PSP30104 Certificate III in GovernmentPSP30104 Certificate III in Government
PSP30104 Certificate III in Government
 
Mensa
MensaMensa
Mensa
 
Reference letter
Reference letterReference letter
Reference letter
 
Reference letter
Reference letterReference letter
Reference letter
 
eSee Talking With Family (instructions for using a web-camera by older people...
eSee Talking With Family (instructions for using a web-camera by older people...eSee Talking With Family (instructions for using a web-camera by older people...
eSee Talking With Family (instructions for using a web-camera by older people...
 
Reference letter
Reference letterReference letter
Reference letter
 
DIAf evaluation survey
DIAf evaluation surveyDIAf evaluation survey
DIAf evaluation survey
 
ICT Central Knowledge Base documentation
ICT Central Knowledge Base documentationICT Central Knowledge Base documentation
ICT Central Knowledge Base documentation
 
How to use the HEAT helpdesk call-logging software
How to use the HEAT helpdesk call-logging softwareHow to use the HEAT helpdesk call-logging software
How to use the HEAT helpdesk call-logging software
 
Information Management System presentation, Office For The Ageing
Information Management System presentation, Office For The AgeingInformation Management System presentation, Office For The Ageing
Information Management System presentation, Office For The Ageing
 
Comparative analysis for J&amp;PS ICT first level helpdesk
Comparative analysis for J&amp;PS ICT first level helpdeskComparative analysis for J&amp;PS ICT first level helpdesk
Comparative analysis for J&amp;PS ICT first level helpdesk
 

Evaluation Guide

  • 1. Workforce Development Evaluation Guide Version: 2012.03.19 Last released on: 17/04/2012 Last released by: Dain Sanyë Senior Consultant Evaluation and Continuous Improvement
  • 2. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Document title Evaluation Guide Document information This document was produced for the Government of South Australia (SA), Department for Education and Child Development (DECD), Human Resources and Workforce Development. This document was created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement for Workforce Development, using Microsoft Word 2010. Purpose This guide describes the evaluation framework created for the Department’s Workforce Development directorate. Audience This document is designed as a guide for Workforce Development officers and managers to effectively evaluate and report on projects and programs. Evaluators should be familiar with:  the layout of a personal computer and desktop  the purpose, creation and analysis of data and survey tools  the Department’s Improvement and Accountability framework (DIAf) This guide is written in international English. Contact details Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Phone: (08) 820 41402 Fax: (08) 8206 4200 Email: Dain.Sanye3@sa.gov.au Copyright notice Copyright ©2012 Government of South Australia, Department for Education and Child Development. This publication is copyright and contains information which is the property of the Department for Education and Child Development. No part of this document may be copied or stored in a retrieval system without the written permission of the author or the Chief Executive of the Department for Education and Child Development. Page EV–2 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 3. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Related files  Checkbox survey tool < http://www.decssurveys.sa.edu.au/online/ >  DIAf < http://www.decd.sa.gov.au/quality/pages/quality/26420/ > Update plan Updates are the responsibility of the current Workforce Development Senior Consultant, Evaluation and Continuous Improvement. Updates must occur when scheduled as a part of ongoing role responsibility or as requested by a Workforce Development line manager. Updates must include an incrementally increased version number in the format yyyy.mm.dd.A (year.month.date.draft version of update). Old versions must be moved to an archive location. The current version must maintain the same file name and be stored and accessible from a single fixed network location. Shortcuts should be created at additional locations linking to the single fixed network storage location. Revision history Please destroy any printed copies of this document earlier than version 2012.03.19. Version Reviser Details 2012.02.22.A Dain Sanyë Initial draft 2012.03.19 Dain Sanyë Initial document released Page EV–3 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 4. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Contents Document title ..................................................................................................... 2 Document information ......................................................................................... 2 Purpose ............................................................................................................... 2 Audience ............................................................................................................. 2 Contact details ..................................................................................................... 2 Copyright notice................................................................................................... 2 Related files......................................................................................................... 3 Update plan ......................................................................................................... 3 Revision history ................................................................................................... 3 Contents .................................................................................................................. 4 1.0 Introduction.................................................................................................. 7 2.0 Background.................................................................................................. 8 3.0 Evaluative review of a workgroup or project team .................................... 9 DIAf corporate self-review snapshot survey ......................................................... 9 Improvement principles...................................................................................... 10 Improvement principle criteria ............................................................................ 10 Focus on core business ............................................................................... 10 Think systemically........................................................................................ 11 Share leadership.......................................................................................... 11 Attend to culture .......................................................................................... 11 Listen and respond ...................................................................................... 11 Make data count .......................................................................................... 12 Set directions ............................................................................................... 12 Target resources.......................................................................................... 12 Continuously improve .................................................................................. 12 Principle analysis ............................................................................................... 13 Aggregate score .......................................................................................... 13 Graphing foci scores .................................................................................... 14 Collaborative discussion .................................................................................... 15 3.A Appendix: Example—DIAf corporate self review survey ........................ 16 4.0 Strategic evaluation of a project ............................................................... 32 Objectives.......................................................................................................... 33 Deliverables....................................................................................................... 34 Strategic plans and alignment............................................................................ 35 Evaluation measures ......................................................................................... 36 Focus areas....................................................................................................... 37 Focusing on development or improvement .................................................. 37 Designed and implemented with systemic thinking ...................................... 37 Enables individual leadership....................................................................... 37 Attentive to organisational culture while transforming capacity..................... 37 Managed stakeholders through effective listening and responsiveness ....... 37 Effectively collated, used and reported data................................................. 37 Page EV–4 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 5. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Maintained direction and scope throughout the project ................................ 37 Innovatively and effectively aligned resources ............................................. 38 Building a sustainable future including continuous improvement.................. 38 Scoring .............................................................................................................. 38 Undeveloped ............................................................................................... 38 Developing................................................................................................... 38 Functioning .................................................................................................. 38 Strategic ...................................................................................................... 38 Embedded ................................................................................................... 38 4.A Appendix: Example—strategic project evaluation .................................. 39 5.0 Evaluation of an ongoing program, with a focus on knowledge and skills transfer .................................................................................................... 48 Level 1—evaluation of motivation and reaction .................................................. 49 Level 2—evaluation of learning content ............................................................. 50 Level 3—evaluation of performance and behaviour ........................................... 51 Level 4—evaluation of impact and results ......................................................... 52 A.0 Appendix: Level 1 sample survey questions ........................................... 54 A.1 Level 1—About the training needs analysis............................................ 54 A.2 Level 1—About the training .................................................................... 55 A.3 Level 1—About the participant ............................................................... 56 A.4 Level 1—Individual learning needs......................................................... 59 A.5 Level 1—Business needs ....................................................................... 62 A.6 Level 1—Core skill needs ...................................................................... 64 B.0 Appendix: Level 2 sample survey questions ........................................... 68 B.1 Level 2—Participant reaction and learning outcomes introduction.......... 68 B.2 Level 2—General feedback .................................................................... 69 B.3 Level 2—Training methods and materials .............................................. 70 B.4 Level 2—Trainer(s) ................................................................................ 72 B.5 Level 2—Tests and qualifications ........................................................... 73 B.6 Level 2—Progress to other learning ....................................................... 73 B.7 Level 2—Pre-training activities and instructions ..................................... 74 B.8 Level 2—Facilities, courses and resources ............................................ 75 B.9 Level 2—Facilities and administration .................................................... 76 C.0 Appendix: Level 3 sample survey questions ........................................... 77 C.1 Level 3—Job performance impact .......................................................... 77 C.2 Level 3—Relevance of the training......................................................... 78 C.3 Level 3—Application of learning ............................................................. 80 C.4 Level 3—Post-training skills observation ................................................ 84 C.5 Level 3—Core skills improvement .......................................................... 86 C.6 Level 3—Job-specific skills evaluation ................................................... 90 C.7 Level 3—Training objectives .................................................................. 91 D.0 Appendix: Level 4 sample survey questions ........................................... 92 D.1 Level 4—Business impact ...................................................................... 92 Page EV–5 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 6. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 D.2 Level 4—Test results ............................................................................. 93 D.3 Level 4—Learning gain .......................................................................... 94 D.4 Level 4—Skills gain ................................................................................ 95 D.5 Level 4—Changes to business performance .......................................... 96 D.6 Level 4—Business performance/impact measures (fixed and open) ...... 97 D.7 Level 4—Financial impact (including ROTI) ........................................... 99 E.0 Appendix: Additional survey questions ................................................. 100 E.1 Open question modifiers and extensions.............................................. 100 E.2 Knowledge and skills tests (question examples) .................................. 101 F.0 Frequently asked questions (FAQs) ....................................................... 103 F.1 How do I get a user account to create surveys in Checkbox ................ 103 G.0 Glossary ................................................................................................... 104 G.1 Definitions ............................................................................................ 104 G.2 Formatting convention.......................................................................... 104 G.3 Colour coding ....................................................................................... 104 G.4 Hyperlinks ............................................................................................ 104 G.5 Acronyms ............................................................................................. 105 Page EV–6 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 7. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 1.0 Introduction Evaluation is defined as the act of appraising to assess value.1 Evaluation identifies and promotes best practice in planning and implementation methodology, and ensures the delivery and creation of Workforce Development products and services is performed at the most efficient and productive level. Effective evaluation of projects and programs ensures consistent, high quality services and products with strong alignment to strategic goals and objectives for all teams’ work across the Workforce Development Directorate. Evaluation also assists in the expansion and roll-over of projects to create sustainable programs by providing a clear and focussed relationship between deliverables, objectives and strategic goals as targeted outcomes and responsibilities in the management process. Evaluation utilises descriptive foci to assist the evaluator to get into a specific frame- of-mind and effectively evaluate the project or program from a variety of strategic viewpoints. Project and program managers are most likely to evaluate their own projects and programs. Evaluative foci have been described from the Department’s Improvement and Accountability framework (DIAf)2. Some of the foci this guide will assist you to evaluate against include your project or program’s:  attention to organisational culture  effective collation, use and reporting of data  effective use of human, financial and physical resources  embedding of leadership and responsibility  focus on development and continuous improvement  maintenance of scope and direction  stakeholder management  sustainability over the medium to long term  systemic thinking and integration The three evaluative processes described in this guide are designed to assist officers and managers strategically prepare and evaluate projects and programs, including:  Evaluative review of a workgroup or project team  Strategic evaluation of a project  Evaluation of an ongoing program, with a focus on knowledge and skills transfer 1 http://dictionary.reference.com/browse/evaluation?s=t 2 www.decd.sa.gov.au/quality/pages/quality/26420/ Page EV–7 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 8. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 2.0 Background Prior to 2012 the Workforce Development directorate did not have a consistent framework to evaluate the Directorate’s projects and programs. Individual teams within the Directorate use a number of different and differently applied evaluation tools, historical or mandatory evaluative processes without scheduled or regular continuous improvement analysis of their evaluative processes, effectiveness and outcomes. The Department’s enterprise Registered Training Organisation (RTO), Organisation and Professional Development Services (OPDS), use:  national VET surveys for participants and line managers of participants  AQTF compliance requirements The Directorate’s Quality Leadership programs use:  The Directorate’s Teacher Quality projects and programs use:  The Directorate’s Performance and Development projects and programs use:  The Directorate’s Projects and Innovations projects and programs use:  Page EV–8 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 9. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 3.0 Evaluative review of a workgroup or project team An evaluative review of a workgroup or project team is recommended once per year; however, an evaluation may be prudent following any significant change or development within the team; for example, new leadership, new location, change in organisational strategic plan, or prior to the planning phase of a significant project. The evaluation is designed as a self-review to identify and promote collaborative discussion on opportunities for continuous improvement and increased effectiveness that aim to improve the productivity, efficiency and capacity across the team. Analysis of the survey responses provides specific direction to improve team engagement, satisfaction, focus, responsiveness and resource use. The evaluation encompasses the Department’s nine DIAf improvement principles. DIAf corporate self-review snapshot survey This survey is stored within the Department’s Checkbox survey tool under the following URL < http://www.decssurveys.sa.edu.au/online/selfreveiw.aspx > and is accessible through the internet to all Directorate officers. The survey is password protected with the current password “opds”. The survey details key criteria to consider as a lead into self-review of any team's performance and operations. Team members should complete the survey individually and encouraged to be honest, noting all responses are anonymous3. After all team members have been given sufficient time to comfortably respond4, the responses should be collated to provide data for collaborative discussion. All the survey's focus questions are answered on a ratings scale from Strongly Disagree to Strongly Agree. This provides an opportunity within the principle analysis to index the team’s aggregate score and accurately report on the team’s improvement trend and recommended foci for collaborative discussion5. There are no text or long answer questions in the survey; these types of comments should be raised in the collaborative discussion following the survey's completion and principle analysis6. After the survey is closed it is important to allow and allocate adequate time for a collaborative discussion with all team members present at a team meeting or other dedicated meeting time. At the collaborative discussion all members of the team are encouraged to discuss the Focus on core business and two other focus principles identified through the principle response analysis as the greatest opportunities to improve effectiveness within the team. 3 Preview and analysis of the survey responses should not begin until after a sufficient number of responses have been received to ensure anonymity for the respondents. 4 As the survey is accessible over the internet, some team members may feel more comfortable completing the survey out-of-office, after hours or at home. 5 Indices should not be compared between teams; however, all staff within the Directorate can be anonymously surveyed at the same time to achieve a current Directorate-level index. 6 Analysis of survey responses should be carried out by an independent and skilled analytical or evaluative Directorate officer; for example, the Directorate’s Data Management and Analysis Officer or Senior Consultant, Evaluation and Continuous Improvement. Page EV–9 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 10. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 The collaborative discussion should encompass: a) evidence used by team members to score the focus questions b) what the next steps would be for improvement and who will be responsible for managing implementation7 c) ongoing self-review processes including evaluation of the implementation of the continuous improvement project d) the data and other evidence that would be gathered to monitor and evaluate improvement over time8 Improvement principles The nine DIAf improvement principles are: 1. Focus on core business 2. Think systemically 3. Share leadership 4. Attend to culture 5. Listen and respond 6. Make data count 7. Set directions 8. Target resources 9. Continuously improve Improvement principle criteria For each improvement principle there are four criteria9 that can be scored on a ratings scale from Strongly Agree to Strongly Disagree by each individual team member. Focus on core business 1.1 Team goals are clear, known and used to drive decisions 1.2 The team has high service and delivery standards that result in high quality outcomes 1.3 Team members are committed to the team’s goals 1.4 The team’s plans, processes and practices work effectively to support team members to achieve goals 7 Implementation of continuous improvement should be managed using the project management framework laid out in the Directorate’s Information Management System (IMS) and using the PDSA (plan, do, study, act) continuous improvement framework cycle. 8 See the chapter in this Evaluation Guide on strategically evaluating a project. 9 Note that principle criteria contain more than one question that may cause and result in skewing of responses where a respondent may agree with part of the criteria and disagree with another part. If this issue is raised, respondents should score the criteria with their lowest desired response rating to ensure the improvement principle has the best opportunity to be discussed for improvement by the team. Page EV–10 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 11. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Think systemically 2.1 Political, system and contextual issues are identified and strategically addressed in plans and practices 2.2 Effective research and development processes enable team members to improve operations, outcomes and service delivery 2.3 Internal management processes are routinely reviewed to continuously improve operations 2.4 Effective partnerships exist with key stakeholders, other business units and professional groups to support the achievement of unit goals Share leadership 3.1 Leaders provide clear direction and supportive leadership, and take an effective stance appropriate to the individual/situation to achieve agreed outcomes 3.2 Leadership is shared with strategies and processes to build the leadership capacity of individuals and leadership density of the business unit 3.3 Leaders support effective business unit management through a focus on professional learning for team members and themselves 3.4 Leaders ensure change is managed positively and successfully, with workload balance and direction sustained Attend to culture 4.1 A positive workplace culture supports team members to work with enthusiasm, commitment, energy and the business unit to achieve success 4.2 Team members’ roles and responsibilities are clearly known and professional team interactions optimise success 4.3 Professional development and performance management processes provide team members with recognition, support and feedback to develop expertise 4.4 Culture and morale building processes effectively support positive team member interactions and address issues and concerns Listen and respond 5.1 Quality partnerships are deliberately developed with key clients and stakeholders to achieve outcomes 5.2 Communication processes provide information to and from clients and stakeholders to improve service delivery and outcomes 5.3 Decision making structures are effective with high levels of team member and stakeholder input, support for, and engagement in decisions 5.4 Commitment to quality service delivery and responsiveness by all team members provides high levels of satisfaction and positive client relationships Page EV–11 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 12. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Make data count 6.1 Effective data management processes are in place to collect, store and access reliable data 6.2 Multiple measures of data are analysed and used to inform improvement directions, evaluate programs and report on outcomes 6.3 Data is used to identify root causes and variation to targeted improvement efforts while monitoring the effectiveness of implementation strategies 6.4 Data creates knowledge and learning for the team, organisation and system to inform decisions on development and innovation Set directions 7.1 An explicitly stated vision, values and purpose developed in collaboration with team members drives business unit decisions, plans and directions 7.2 Planning processes build team members’ capacity and expertise to achieve the vision and continuously improve outcomes 7.3 Communication, monitoring and evaluation of planning processes occurs with high levels of team member involvement and ownership 7.4 Strategic plans are integrated and enacted in daily operations to ensure strategic directions are achieved Target resources 8.1 Effective resource management systems identify, support and develop the team’s human, financial and physical resources 8.2 Resources are targeted to achieve successful outcomes with processes in place to review resource needs and effectiveness 8.3 Assets and resources are acquired, organised and maintained to support performance 8.4 Risk management processes ensure prudent financial management, regulatory compliance and safe workplace practices Continuously improve 9.1 Effective, known improvement processes support team members to ensure that goals are achieved and outcomes are continuously improved 9.2 Rigorous, regular self-review processes occur, with team members involvement and engagement, to monitor outcomes, evaluate progress and inform future directions 9.3 A commitment to continuous improvement is evidenced by routine policy review and development cycles, and effective document and records management 9.4 The team’s members develop integrated, sustainable and systemic programs, projects and products to achieve business unit goals and respond to the emerging needs of stakeholders Page EV–12 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 13. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Principle analysis The principle analysis of the survey’s responses should be calculated per improvement principle focus and include:  an aggregate score of the responses to the criteria for each improvement principle (focus)  the total number of responses and total number of possible responses (team members)  a graph displaying the distribution of relative scores across the focus and its four criteria independently Aggregate score The aggregate score may:  remove outliers—where the aggregate of one rating score is significantly inconsistent with the median or mode rating score (eg out of 25 responses: one response Strongly Disagrees while 24 responses Agree or Strongly Agree; the Strongly Disagree response may be removed from the analysis)10  be reported as a definitive modal score11—if the modal score aggregate exceeds the sum aggregate of all other rating scores (eg out of 25 responses: 16 responses are Agree while 9 responses are other ratings; the aggregate score for this focus can be reported as Agree)  be reported as a definitive median score12—if the aggregate scores per rating appear as a balanced bell curve (eg out of 25 responses: 1 Strongly Disagree, 3 Disagree, 6 Neither, 9 Agree, 6 Strongly Agree; the median score for this focus can be reported as Agree)  be reported as a mean or average index—with each rating given an linear aggregate multiplier (eg out of the above bell curve distribution with an aggregate multiplier of x1) Rating Strongly Disagree Disagree Neither Agree Strongly Agree (Multiplier) (x1) (x2) (x3) (x4) (x5) TOTAL Score 1 3 6 9 6 Aggregate 1 6 18 36 30 91 Possible maximum 125 INDEX = 3.64 Agree (72.8%) The aggregate score for each improvement principle focus should always be calculated and reported consistently. 10 If there is a significant cause to the response removed from the analysis this is expected to be raised during the collaborative discussion. 11 The modal score is the distinct rating with the highest aggregate. 12 The median score is the exact middle rating when all scores are accurately ordered from lowest to highest. Page EV–13 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 14. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Graphing foci scores The distributive graph should be chosen and produced to provide a highly visual discussion prompt for the collaborative discussion; for example: Page EV–14 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 15. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Collaborative discussion The collaborative discussion is the most important part of the evaluation. The discussion enables team members to focus on specific areas for improvement, identify barriers and work as a team to trouble-shoot solutions to issues that impact the self-identified effectiveness of the team. Only three foci should be discussed in this meeting including the Focus on core business plus two other foci recommended by the independent principle analyst as areas which have a very low aggregate score or where the aggregate score does not appear representative of a clear majority; (eg out of 25 responses: 12 Strongly Disagree, 12 Agree, 1 Strongly Agree). The collaborative discussion should occur as a dedicated meeting with no other agenda items, in a closed meeting room set up for brainstorming (whiteboards, butcher’s paper, post-it notes), and all team members present. If possible the discussion should be led by an independent facilitator. The meeting must allow for anonymous reporting of causative factors leading to the opportunity for improvement; (eg all team members may be asked to write anonymously a possible cause for the low criteria or focus score on a post-it note and pass those up to the independent facilitator who will ensure no individuals are identified while the causes are being discussed). The facilitator should use techniques like the 5-Why’s13 to determine root causes of the score and to provoke collaborative discussion on potential resolutions. After resolutions have been brainstormed, the facilitator should lead the team in identifying the resolution project to be recommended The project proposal should be development through project management framework, with the team brainstorming components including:  potential risks  implementation schedule  stakeholders to be engaged  project team and responsibilities  objectives and deliverables to be achieved  evaluation process to ensure completion  and measurable, reportable achievement14 The improvement project manager or team should report the improvement project’s ongoing status back to the whole team as an agenda item at every future normal team meeting until the project is closed. 13 To identify the root cause of an issue, ask Why the issue occurred, then ask Why the cause of the issue occurred, then ask Why the cause of the cause of the issue occurred, to a total of 5-Why’s. 14 See the chapter in this Evaluation Guide on strategically evaluating a project. Page EV–15 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 16. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 3.A Appendix: Example—DIAf corporate self review survey Page EV–16 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 17. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–17 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 18. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–18 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 19. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–19 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 20. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–20 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 21. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–21 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 22. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–22 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 23. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–23 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 24. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–24 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 25. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–25 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 26. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–26 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 27. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–27 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 28. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–28 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 29. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–29 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 30. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–30 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 31. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–31 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 32. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 4.0 Strategic evaluation of a project A project is a temporary measure with distinct start and end dates comprising a time constraint to meet specific goals and objectives, and produce deliverables within a defined budget and limited resources, in order to achieve beneficial, value-adding change.15 A program does not have a defined end date and usually contains multiple repetitious project instances defined for example, per annum.16 Projects and programs should be evaluated by the project or program manager, or a specified third party, during the:  project concept or proposal phase  throughout the project implementation as required  post-project review or closure phase Strategic evaluation of a project or program enables evaluation against its:  Objectives  Deliverables  Strategic alignment  Evaluation measures Project objectives align to strategic plans and goals, and are achieved and evidenced through the production of project deliverables throughout the implementation of the project. The project’s deliverables are evaluated to ensure they have effectively achieved the project’s objectives and strategic alignment through evidence of achievement. This evaluation framework assist the project or program manager evaluate their project’s deliverables, objectives, strategic alignment and evaluative processes against foci developed from the Department’s Improvement and Accountability framework (DIAf). All Workforce Development projects and programs including courses and events should be recorded in the Directorate’s information management system (IMS). < decsgla01user3GroupsHRWorkforce DevelopmentIMSinterfacesProject Management.mdb > This evaluation framework is planned to be integrated into the IMS Work Program and Project Management interface. 15 For example, the 2012 South Australian Public Teaching Awards is a project. 16 For example, the South Australian Public Teaching Awards is a program. Page EV–32 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 33. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Objectives Objectives to be achieved through the project should be recorded into the IMS. An objective is a key benefit achievable by the project that aims to meet a strategic direction, plan or vision for the Department. Objectives are usually described in the format: To..., in a way that..., so that... Objectives have no physical substance; they are not created in a visible way.17 Score Description Example Undeveloped objectives resemble a brain-storming To improve image list, are vague and have no indication of change or outcomes Developing objectives are stated broadly and infer To improve the image of the or imply non-specific outcomes or Department changes in practice Functioning objectives are clear about the To improve the image teachers have outcomes to be targeted or improved of the Department recognising them but the change is stated in broad non- as part of the workforce specific terms Strategic objectives are linked to a strategic To improve the image teachers have objective in a plan and expressed in of the Department recognising them terms of the outcomes to be targeted as an important part of the education or improved and care workforce Embedded objectives are STRATEGIC, based on To improve the image teachers have best practice, aligned to increase of the Department recognising them effectiveness and consistency, and as the part of the education embedded with a specific objective in workforce that provides excellence a strategic plan in education and care Please update the objectives of your project in the IMS before scoring. 17 Compare with deliverables. Page EV–33 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 34. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Deliverables Deliverables to be created during the project’s implementation should be recorded in the IMS. A deliverable is a key product that will be created by the project providing physical evidence of the project's activity.18 Deliverables include key performance indicators (KPIs) as statistics and are used as proof the project is meeting its Objectives. Score Description Example Undeveloped are when no deliverables are None stated and implementation does not produce any physical evidence Developing deliverables are common or Hold an awards ceremony for 2012 historically repeated with no significant improvement evident Functioning deliverables include data Hold a ceremony for 100 award winners in related to the project but 2012 lacking specificity against changes or improvements to be achieved Strategic deliverables are assigned to The Workforce Recognition Officer will event specific team members, are manage the 2012 SA Public Teaching demonstrable including and Awards ceremony for up to 100 teachers supported by data, and recognised and nominated in the last 12 expressed as SMART months for outstanding teaching practice (specific, measurable, achievable, relevant, timely) Embedded deliverables are STRATEGIC, The Workforce Recognition Officer will event and include agreements for manage the 2012 SA Public Teaching continuous support and Awards ceremony promoting excellence in improvement with deliverables education and care for up to 100 teachers for post-project analysis and recognised and nominated in the last 12 review months for outstanding teaching practice using a documented up-to-date event management process including a survey of participants on the event's delivery Please update the planned and achieved deliverables of your project in the IMS before scoring. 18 Compare with objectives. Page EV–34 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 35. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Strategic plans and alignment A link from each project objective to a specific strategic plan’s objective or goal should be recorded against each project objective in the IMS. Linking objectives to a strategic direction, plan or vision is an important analysis of the value of your project. Score Description Example Undeveloped There are a high number of competing strategies which are very broad, are management or compliance requirements, or are determined by the Department Developing There are a range of broad strategies areas, based on complex targets that are a mix of management and improvement issues, already described by the Department Functioning There are a number of strategies that are mainly improvements or developments, identified from strategic planning, Departmental or national objectives Strategic There are a number of strategies that clearly define continuous improvement integrated with Departmental or national objectives, determined through data analysis, agreed to by staff and described against a particular field of expertise Embedded There are a number of strategies that all staff agree from data analysis and stakeholder consultation, that are key objectives to focus for continuous improvement, that expressed against particular fields of expertise will concurrently and strategically achieve Departmental or national objectives Please update your project's objective links to strategic plans in the IMS before scoring. Page EV–35 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 36. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Evaluation measures Projects managed through the IMS have measures enabling evaluation to be described and resulted through your project's objectives, deliverables and strategic links. Evaluation includes your project's ongoing status updates as well as the final project review closure report. Score Description Example Undeveloped Data may be collated but is not used to monitor # participants progress, support status reporting or evaluate the effectiveness of the project Developing Measure of progress are merely numbers or gender distribution of # deliverables ticked as completed or achieved with participants minimal or no evaluation of the relative change or survey of participants improvement achieved interest Functioning Data is used throughout the project to monitor rate of registration of and report on the progress and achievement of interest targets, deliverables and objectives against strategic plans Strategic Specific data is regularly collated and used to regional and index of monitor the progress of the continuous disadvantage improvement changes throughout the project's distributions of rate of lifecycle, with regular review of the completion of registration and deliverables to support the achievement of achievement vs student objectives against strategic plans by all project outcomes (NAPLAN) team members Embedded Evaluation is STRATEGIC with the analysis of post event survey and multiple measures of improvement and the data retrieval regular collaboration and input from all project quantitatively analysing team members to evaluate the project's progress changes and and refine and redefine the objectives and next improvement in steps towards the most effective deliverables to education and care achieve the strategic outcomes of the project activity Please start your post project review in the IMS before scoring, including: • what worked • what didn't work • lessons learned • your recommendations for future project managers Page EV–36 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 37. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Focus areas The following focus areas may be used to categorise the project’s objectives, deliverables, strategic alignment and evaluation measures for projects where the implementation of the evidentiary deliverables to achieve the project objectives aligned with strategic plans is significantly large or complex. Focusing on development or improvement The project produced evidence of continuous improvement in achievement/outcomes for stakeholders, with challenging targets for ongoing improvement and quality practice, while creating an ethos with high expectations and consistent understanding to drive, develop and improve policy, practice and performance. Designed and implemented with systemic thinking The project is integrated with wider systems and is committed to improve these systems to create an aligned and effective integrated system that supports the Department's continual improvement with targets for achievement appropriate to the context of community needs and aspirations. Enables individual leadership The project develops leadership within the project's team members and stakeholders enabling them to exhibit principled and visible responsibility that is shared and fostered throughout the project through effective project management expertise and capacity at all levels to achieve the project's objectives, deliverables and strategic goals. Attentive to organisational culture while transforming capacity The project intentionally creates a culture that involves individuals and groups in transforming the capacity of the system through a positive culture with high levels of team and stakeholder satisfaction, morale and support for individuals to grow and improve their performance. Managed stakeholders through effective listening and responsiveness The project enables team members and stakeholders to have an active voice and be responsive to current and future needs of the Department using a client-focussed approach to communication, risk management, prioritisation, and organisation of the project's components and responsibilities. Effectively collated, used and reported data The project creates or gathers the necessary information and knowledge required from data sources including stakeholder needs to strategically evaluate and improve outcomes through an informed, structured and organised approach with a clear evidence base for decision-making and recommendations as well as identifying clear drivers for continuous improvement opportunities. Maintained direction and scope throughout the project The project planning process, documentation and stakeholder communication explicitly states values, vision and purpose within its objectives, deliverables and strategic links, and all documentation is consistently up-to-date and available to all project team members. Page EV–37 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 38. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Innovatively and effectively aligned resources The project innovatively targets resources to most effectively align all physical, human and financial resources with project needs through sustainable resource management to achieve successful and efficient outcomes. Building a sustainable future including continuous improvement The project achieves improvement for the Department through a structured continuous improvement cycle with regular evaluation and review of existing related processes and activity. Scoring Scoring of your project’s objectives, deliverables, alignment to strategic plans, and evaluation measures is recorded as a single rating across each component either as a whole, or categorised into focus areas. 1. Undeveloped 2. Developing 3. Functioning 4. Strategic 5. Embedded Undeveloped An undeveloped rating refers to objectives, deliverables, strategic alignment and evaluation measures that are undocumented, unrecorded and unable to be reviewed by an appropriate third party. Developing A developing rating refers to objectives, deliverables, strategic alignment and evaluation measures that are drafted, or only documented on the initial project plan, concept or proposal. Functioning A functioning rating refers to objectives, deliverables, strategic alignment and evaluation measures that have been or are intended to be ticked off as achieved or not achieved; however, the project’s objectives or deliverables are not linked, related or aligned to strategic plans, goals or objectives. Strategic A strategic rating refers to objectives, deliverables, strategic alignment and evaluation measures where the project’s objectives and deliverables are linked, related, aligned and described against strategic plans, goals or objectives. Embedded An embedded rating refers to objectives, deliverables, strategic alignment and evaluation measures where the project only includes objectives that are linked to strategic objectives; with all project objectives evidenced by deliverables (with no non-strategic, orphan objectives or deliverables); and with the project plan, implementation, delivery and closure evaluation focused on strategic achievement. Page EV–38 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 39. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 4.A Appendix: Example—strategic project evaluation Page EV–39 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 40. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–40 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 41. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–41 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 42. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–42 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 43. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–43 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 44. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–44 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 45. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–45 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 46. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–46 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 47. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Page EV–47 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 48. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 5.0 Evaluation of an ongoing program, with a focus on knowledge and skills transfer This evaluation is designed to encompass both effective training (improvement of programs) and training effectiveness (implementation of skills to improve organisational effectiveness). The Kirkpatrick19 model will be used as the structural framework in the evaluation of Workforce Development training programs. This model has the following four levels. Level 1—evaluation of feedback from participants and stakeholders Level 2—evaluation of physical content, resources and deliverables Level 3—evaluation of application and change to organisational processes Level 4—evaluation of strategic outcomes and measurable benefits Levels 1–2 provide instructive feedback to program deliverers to update and improve their relationship with participants, and to project managers in the roll-over of project initiatives to ongoing programs for the future program manager. Levels 3–4 produce summation of strategic value and effectiveness in the implementation of a project and ongoing program delivery reportable to senior management. Some methods of evaluation include:  asking for self-evaluation from participants and facilitators  testing participants’ knowledge, understanding and decision-making skill  observing participants’ performance  examining organisational business results  comparing social media learning with traditional learning intervention  seeing how much productivity is lost or gained from time required Results from evaluations can be analysed in global categories to calculate the effectiveness of a type of training for the organisation; for example, all leadership program evaluations. Sample evaluation survey questions are listed in Appendices A–E. 19 www.kirkpatrickpartners.com Page EV–48 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 49. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Level 1—evaluation of motivation and reaction The first level of evaluation is used to determine how well your participants engage with the registration and learning process. It is used to identify why participants have enrolled in the program, what they perceive are needs in their workplace that the program will have a positive impact on to improve their working environment, and what their learning expectations are from participation in the program. The motivation and reaction evaluation informs on how appealing, relevant and effective the learning content and delivery is to the individual participants, measuring how well the learning engagement processes and program reputation work. Even if all the other levels are effectively covered by the program, missing level 1 can cause participants to fail to see a purpose in the learning and they may disengage from the program. Effective evaluation of level 1 motivation and reaction to the program to measure the engagement of participants to its structure and marketing can be collated and analysed through:  reaction sheets  surveys  focus groups  interviews Participants may self-assess their impression of the program pre-, during and post- participation against any improvement rating scale with the following foci:  relevance  specific  practical  accessible  social Page EV–49 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 50. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Level 2—evaluation of learning content The second level of evaluation is used to improve the quality and effectiveness of the learning content, structure and delivery. It is used to update, refine and ensure best practice in the knowledge, skills, and resources provided to and developed by participants who participate in the facilitator’s program. The learning content evaluation informs on what participants learned and the extent to which knowledge and skills were gained by the participants, including the degree of effectiveness that the learning delivery achieved to transfer the content, knowledge and skills to the participants. Even if all the other levels are effectively covered by the program, missing level 2 can cause the participants to devalue the learning and consider the program to be out-of-date and irrelevant. Effective evaluation of level 2 learning content in the program to measure successful content design and development, delivery methods and achievement can be collated and analysed through:  pre and post training results  written knowledge tests  role-play and simulation  activities and games Page EV–50 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 51. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Level 3—evaluation of performance and behaviour The third level of evaluation is used to report on changes and increases in capacity of job performance resulting from participation in the program, driving accountability, measuring effectiveness and value to the organisation, and enabling appropriate resource and support allocation. It is used to prove value of participation in the program through identifying changes in the way employees behave and perform in their working environment to increase the efficiency, productivity and quality of their work using newly learned skills and knowledge. The performance and behaviour evaluation informs of the capability of participants to effectively uptake and perform newly developed skills, and the degree that learned skills and knowledge actually transfer to and are used in the working environment, post-participation. Even if all the other levels are effectively covered by the program, missing level 3 can cause the program to be devalued by staff and management as they consider the skills and knowledge learned through participation to be academic and ineffective in the real workplace. Effective evaluation of level 3 performance and behaviour following participation in the program to measure successful transfer and implementation of learning from the training to working environments, can be collated and analysed through:  surveys  interviews  focus groups  observations  work reviews Participants may self-assess changes in their working environment following participation in the program against any improvement rating scale with the following foci:  performance  confidence  contribution  engagement  retention  customer satisfaction  business success  systems use  drivers and strategic alignment Page EV–51 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 52. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 Level 4—evaluation of impact and results The fourth level of evaluation is used to report on the tangible organisational outcomes achieved through the implementation of skills developed by participation in the program. It is used to graphically demonstrate the productive impact, outcomes and results of participation showing organisational improvement. The impact and results evaluation provides quantitative results and scoring of added organisational value including reduced costs, improved quality, increased production and efficiency indices, and enables calculation of return on investment (ROI) from the program. Even if all the other levels are effectively covered by the program, missing level 4 can cause the program to have reduced priority and acknowledgement by C-level and Executive management as they may not easily identify a positive impact on the organisation’s bottom line by supporting the program. Effective evaluation of level 4 impact and results to measure the outcomes of implementation of learning to achieve described targets can be collated and analysed through:  borrowed metrics from other data systems including human resources data  surveys  focus groups Decision-makers prefer the results of level 4 evaluations, although not necessarily in dollars and cents. For example; a study of financial and information technology executives found that they consider both hard and soft returns when it comes to customer-centric technologies, but give more weight to non-financial metrics (soft), such as customer satisfaction and loyalty.20 These evaluation results enable real business results to be connected to training programs, including:  operational efficiency  compliance  retention of top talent  customer satisfaction  sales volume These results also identify through analysis, the gaps and needs of the organisation that can be filled through training and skills development. 20 Hayes 2003 Page EV–52 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 53. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 This level of program evaluation may be met with a more balanced approach or scorecard21 from four perspectives:  Financial—a measurement that shows a monetary return or impact such as how the output from a process is improved (financial can be either soft or hard results)  Customer—improving an area in which the organisation differentiates itself from competitors to attract, retain, and deepen relationships with its targeted customers  Internal—achieving excellence by improving processes as supply-chain management, production or support process  Innovation and learning—ensuring learning packages support a climate for organisational change, innovation, and the growth of individuals Evaluative results from the above scorecard are preferred by management but may be supplemented with the more commonly provided levels 1–2 quantitative results listed below.  How many people (participants) will receive the training?  How often the training both can and is planned to be repeated; noting that online training courses can be developed once and re-used with relatively low ongoing costs, whereas the volume of financial, human and physical resources required for classroom delivery continues to increase  How many total hours of learning have been successfully achieved?  A comparison of the cost of initial development (initial or single class), with the reducing cost to maintain and deliver multiple instances of the same content  Overall resource costs involved in preparing, running and evaluating programs, in both dollars ($) and hours.  Do participants and/or facilitators need to travel, be accommodated, and supplemented at the workplace while they are training?  What proportion of programs were evaluated, and the distribution of stringency of those evaluations?  What proportion of programs were evaluated by the program manager, versus those evaluated by an independent third party evaluator?  The effective distribution of evaluation results.  How many programs or instances were planned using or referencing evaluation results from previous programs or relevant projects, before implementation?  What evidence is there of the implementation of evaluation recommendations in the delivery of programs? 21 Kaplan, Norton 2001 Page EV–53 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 54. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 A.0 Appendix: Level 1 sample survey questions The first level of evaluation is used to determine how well your participants engage with the registration and learning process.22 A.1 Level 1—About the training needs analysis 1. This training needs analysis asks about [DELETE AS APPROPRIATE] the business need for training / your individual learning needs and preferences. 2. The assessment is made up of [ADD NUMBER] questions and should take about [ADD NUMBER] minutes to complete. Please answer the questions as fully as possible. 3. Please complete the assessment by [ADD DETAILS]. 4. What will happen with your responses?  [DELETE AS APPROPRIATE]  Your responses will be used to help decide on the most appropriate training needed.  Your responses will be anonymous and feedback will be reported for all respondents in generalised form only.  All responses will be analysed and reported to [ADD DETAILS]. Findings are due to be reported in [ADD DETAILS].  A copy of the assessment report will be available from [ADD LOCATION/PARTICIPANT] on [ADD DATE]. 5. Any questions? 6. If you have any questions about this assessment, please contact [ADD DETAILS]. 22 The following questions have been adapted from < www.trainingcheck.com >. Page EV–54 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 55. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 A.2 Level 1—About the training 1. What was the name of the training/activity you attended? 2. Which training event did you attend? 3. Which sessions did you attend? Choose as many as apply.  All sessions  Session 1  Session 2  Session 3 4. Which modules did you attend?  All  Module 1 only  Module 2 only  Other, please specify 5. How many sessions did you attend in total? 6. Where was the training/activity held? 7. Which location did you attend? 8. What was the date of the training?  dd/mm/yyyy 9. Which date did you attend? 10. Which times did you attend? 11. Which level did you attend? 12. What type of training/activity was it?  Classroom-based  Web-based/e-learning  On-the-job learning  Coaching/mentoring  Project work  Job shadowing  Other, please specify 13. What was/were the name(s) of the trainer(s)? 14. Who was the trainer? Page EV–55 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 56. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 A.3 Level 1—About the participant 1. Your name: 2. Your organisation: 3. Your job title/grade: 4. What is your job title/grade?  [FIRST JOB TITLE / GRADE / CLASSIFICATION]  [SECOND JOB TITLE / GRADE / CLASSIFICATION]  [THIRD JOB TITLE / GRADE / CLASSIFICATION]  Other, please specify 5. Which of these best describes your job role?  Senior manager  Manager  Supervisor  Employee  Other, please specify 6. On what basis are you employed?  Full-time  Part-time  Contract/agency  Other, please specify 7. Which department/section do you work in? 8. What is your manager's name? 9. Which shift pattern do you work in: 10. How long (in months) have you been in your current role? 11. How long have you worked for the organisation?  Less than one year  1-2 Years  3-4 Years  5-10 Years  11-15 Years  16-20 Years  21-25 Years  More than 25 years Page EV–56 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 57. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 12. Your age:  Under 18  18-25  26-35  36-45  46-55  56+ 13. What is your date of birth?  dd/mm/yyyy 14. Your gender:  Male  Female 15. Is English your first language?  Yes  No 16. What is your primary language?  English  [LANGUAGE 2]  [LANGUAGE 3]  [LANGUAGE 4]  Other, please specify 17. What is the highest level of education that you have completed?  Primary/elementary school  Secondary/high/vocational school  Further tertiary education/vocational college  University  Other, please specify 18. What is the highest grade or year of school you completed?  Never attended school or only attended kindergarten  Grades 1 through 7 (Primary)  Grades 8 through 11 (High school)  Grade 12 (High school graduate)  Tertiary 1–3 years (TAFE or University)  Tertiary 4+ years (Post-graduate)  Post-tertiary (Post-graduate Honours, Masters, Doctorate)  Other, please specify Page EV–57 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc
  • 58. Created by Dain Sanyë, Senior Consultant, Evaluation and Continuous Improvement Version 2012.03.19 19. What is your highest qualification?  Doctorate  Master’s  Honours  Bachelor's  Diploma  Certificate  None  Other, please specify Page EV–58 of EV–105 Evaluation—Guide R:HRWorkforce DevelopmentProjects and InnovationsEvaluationContinuousImprovementEvaluationGuide.doc