SlideShare uma empresa Scribd logo
1 de 75
Baixar para ler offline
ROI of Human Capital Interventions

        A Practice Toolkit
Description         Icon

Relevant quote


Concept/Framework


Tools, Techniques


Template
Step 1

Presenting Problem to Real Problem
Step 1. Presenting Problem to Real Problem

 The first task of the practitioner is almost always analysis




                                             Starts with a
                                            “presenting”
                                           problem – not a
                                            “hunch” but a
                                           problem … as it
                                            presents itself
Step 1. Presenting Problem to Real Problem

                                       Example…




                                                                  Let's take an
                                                               example – higher
                                                               staff turnover for
                                                                   Graduate
                                                               Engineers (let say
                                                                compared to an
                                                              industry standard)




The first question should not be, what skills and knowledge are required. It should be which
                              metric(s) are affected negatively.
Step 1. Presenting Problem to Real Problem

                                        Example…



                      Cost to fill a vacant       On boarding &
                            position              Orientation cost




 Cost of 'covering' a
    vacant position
 (Calculated costs of                                                Cost of productivity
   other employees                                                        ramp-up
 'filling in' while the
  position is vacant)                  Unit cost per
                                       replacement
Step 1. Presenting Problem to Real Problem

                        1.1 Calculating benchmark employee cost


Nr   Metric Element                         Description           Example
1.   Departing employee annual base         N/a                   100,000
     salary
2.   Calculated annual benefits cost        Estimated at 20% of   20,000
                                            base salary
3.   Calculated monthly salary + benefits                         10,000
4.   Calculated daily salary + benefits     Based on 230 (8hr)    522
                                            working days
Step 1. Presenting Problem to Real Problem
1.2 Cost of 'covering' a vacant position (Calculated costs of other employees 'filling
                           in' while the position is vacant)



Nr   Metric Element                          Description         Example
1.   Number of days until the vacant         N/a                 90
     position is filled
2.   Calculated daily cost of 'covering' a   33% of departing    522/3
     vacant position                         employee's daily
                                             salary + benefits
3.   Total cost to 'cover' vacant position                       15644
Step 1. Presenting Problem to Real Problem

                                     1.3 Cost to fill a vacant position

Nr   Metric Element                                     Description                  Example

1.   HR/Hiring manager's annual salary                  N/a                          200,000

2.   Calculated HR/Hiring manager's hourly rate         Based on 230 (8hr) working   130
                                                        days and 20% fringe rate
3.   Cost of advertising (online and/or print)          Enter cost                   3000

4.   Cost of resume screening                           HR/Hire manager hours *      50*130=6500
                                                        hourly rate
5.   Cost of interviews (telephone screening, 1st and   HR/Hire manager hours *      50*130=6500
     2nd)                                               hourly rate
6.   Cost of behavioural and skills assessments         Enter cost                   3000
7.   Cost of background checks (criminal, credit,       Enter cost                   4000
     reference, education)
8.   Cost of travel/moving expenses (if applicable)     Enter cost                   5000

9.   Total cost to fill a vacant position:                                           28,000
Step 1. Presenting Problem to Real Problem

                               1.4 On boarding & Orientation cost




Nr   Metric Element                           Description                  Example

1.   Trainer/Manager annual salary:           N/a                          100,000

2.   Calculated trainer/manager daily rate    Based on 230 (8hr) working   522
                                              days and 20% fringe rate
3.   Total training days                      Enter number of days         10

4.   Total on boarding and orientation cost   N/a                          5220
Step 1. Presenting Problem to Real Problem
     1.5 Cost of productivity ramp-up (During the first 3 months, an average new
      employee performs at 50% productivity of a tenured top performing employ




Nr     Metric Element                                  Description                  Example

1.     Daily employee cost (salary + benefits)         N/a                          522

2.     Number of working days during first 3 months:   Enter number of days (avg.   50
                                                       58 days
3.     Cost of productivity ramp-up                    N/a                          26100
Step 1. Presenting Problem to Real Problem

                     1.6 Unit cost for filling a vacancy




                                  R 74964


Two important learning points:
I.   Know your metrics
II.  Negotiate, and even request help from your client to determine the right metrics. They are
     closer to the business and at the same time you obtain by in for the intervention.
Step 1. Presenting Problem to Real Problem

      1.7 Total cost of replacing graduate engineers




                     R 74964 x 6
                     = R 449784




Let's assume that we had to replace five graduate engineers the
                       past six months
Step 1. Presenting Problem to Real Problem

                                     Now we can move on to analysis


              We know now which
              metrics are affected


                                                                Consultant speciality



Presenting          Analysis and                            Identification of
                                      Solution                                                   Intervention
 Problem/           Definition of                             Performance       Competence Gap
                                     Generation                                                    Strategy
Opportunity        Business Need                                  Need




                                                        Non Training
                                                          Solution
Step 1. Presenting Problem to Real Problem

           The basic tasks in analysis



                                   1. Data
                                  gathering
                2. Data
             consolidation




                             3. Data
                             analysis




          4. Perspective and information
                for decision making
Step 1. Presenting Problem to Real Problem

                               Two types of data



                                                   Quantitative

”The truth is in a number” –
however does not say much
about context
                                                      Data
” More about context, how
people experience in their
own words”.

                                                   Qualitative
Step 1. Presenting Problem to Real Problem

                               Data gathering


I. Questionnaires
II. Surveys
III. Tests
IV. Interviews
V. Focus groups
VI. Observation
VII. Performance records
VIII.Knowledge and skills testing
IX. Program follow up
X. Project assignments
Step 1. Presenting Problem to Real Problem

                                    Data gathering

I.      Target population/participants;
II.     Supervisors of target
        population/participants;
III.    Subordinates of target
        population/participants;
IV.     Peer Group;
V.      HRD Staff;
VI.     Documentation:
       • Organizational Performance
            Records:
       • Disciplinary records;
       • Safety records;
       • Scorecards
Step 1. Presenting Problem to Real Problem

       Contextual factors to consider when selecting data collection methods




I.  Time for participants to respond;
II. Costs;
III.Amount of disruption of normal
    activities;
IV. What is the level of certainty that
    you will achieve with the result;
V. Practicality;
VI. Culture/Philosophy – example the
    10% syndrome.
Step 1. Presenting Problem to Real Problem

      Requirements for effective data collection instruments and techniques



I.   Construct validity ( methods,
     techniques and test instruments
     should be evaluated in terms of its
     theoretical grounding);
II. Predictive validity (methods,
     techniques and test instruments
     should be evaluated regularly in
     terms of its ability to predict what it
     purport to predict);
III. Face validity – for line management
     very important!
Step 1. Presenting Problem to Real Problem

     Requirements for effective data collection instruments and techniques




Reliability:
I. Produce consistent results
    over time, that differentiates
    effectively, and are
    independent of the
    assessor/data collector, or
    context.
Step 1. Presenting Problem to Real Problem

                Data consolidation


Data Consolidation Techniques are
        Wide and Varied

                    Descriptive statistics


        Qualitative description, using organizational
              and other models and concepts
Step 1. Presenting Problem to Real Problem

                  Data analysis



 Data Analysis Varies According to
             Situation

           Descriptive and inferential statistics


        Qualitative decisions about the situation,
        based on decision making and techniques
Note the following !
• Because this toolkit does not aim to transfer knowledge about
  analysis as such the previous slides provides a brief and high level
  overview, for the sake of a clearity… for the sake of the toolkit we
  assume that the analysis provided clarity about the problem….
Step 1. Presenting Problem to Real Problem



Brief Exposition of Problem
Our analysis shows that the HR and line
managers involved in selection decisions do
interviews in a haphazard and nonscientific
                                              Lets assume
fashion                                        this based
                                              on the core
Solution                                        example
Following discussions with consultants, HR
experts and line all agreed we need to
implement competency based interviewing
Step 1. Presenting Problem to Real Problem



The training department is instructed to
develop an appropriate program and put all
line and support staff through training – they
however indicate that a change in results is
required quickly and that ROI for the training
need     to   be     justified   soon     after
implementation…
                                                   Lets assume this
                                                  based on the core
                                                       example
Presenting Problem to Real Problem
So we have addressed this even before we asked any questions about training – and
                      now we can move on to the next step




                                                     Starts with a
                                                    “presenting”
                                                   problem – not a
                                                    “hunch” but a
                                                   problem … as it
                                                    presents itself
Step 2a

Planning the Intervention (Starting With the End
                    In Mind)
Step 2a: Planning the Intervention (With ROI in Mind)

When we plan an intervention – considering that we want to measure ROI- we really reverse
  engineer (or start with the end in mind) – We use the Kirkpatrick Model to achieve this
                         4. Impact

                         What is the business impact we need to achieve (refer step 1
                         where we got a firm grip on the metric that must change)


                         3. Transfer

                         What is the work place behaviour that must change


                         2. Learning

                         What knowledge, skills and attitudes need to be acquired


                         1. Reaction

                         What is the required reaction of the learner to the
                         intervention
Step 2a: Planning the Intervention (With ROI in Mind)

Back to our case study to show how we plan “backwards” – remember these objectives and
 measures for clarification, based on our core example/case study (re graduate engineers)

Kirkpatrick Level   Objectives                                               Measures

Impact              Train target population (define) in behaviour based      25% reduction in total cost of filling
                    interviewing techniques, with the aim to:                vacancies for graduate engineers
                    a. Improve predictive validity of selection decisions
                    b. Driving down cost of filling vacancies for graduate
                         engineers
Transfer            Correct application of behaviour based interviewing      Expert observation with the aid of
                    techniques by target populations during selection        checklists show that all involved in
                    interviewing                                             selection interviews use the correct
                                                                             interviewing structure (agenda) and
                                                                             questioning techniques at least 85%
                                                                             of the time. Observation is
                                                                             conducted for two months following
                                                                             the initial training and corrective
                                                                             coaching is implemented where
                                                                             required, following interviews.
Step 2a: Planning the Intervention (With ROI in Mind)

Back to our case study to show how we plan “backwards” – remember these objectives and
                               measures a merely examples

Kirkpatrick Level   Objectives                                               Measures

Learning            Using Blooms Taxonomy of Learning Objectives here are    100% compliance with the Bloom
                    a few examples of the objectives that will be outlined   Taxonomy, alignment with SAQA Unit
                    here:                                                    Standards and also SAQA guidelines
                    a. The learner should demonstrate that they can          and requirements for assessment
                         analyse a CV, with the aim to develop behaviour     and moderation;
                         based questions for an interview;                   Learners score 75% on theoretical
                    b. The learner should demonstrate the ability to         paper to pass and 80% on practical
                         develop interview questions according to the        simulation interview.
                         behavioural interview format
                    c. Etc. etc…



                                                                Remember this is a
                                                                snapshot – there is a
                                                                whole learning map and
                                                                course development
                                                                process underlying
                                                                these objectives
Step 2a: Planning the Intervention (With ROI in Mind)

Back to our case study to show how we plan “backwards” – remember these objectives and
                               measures a merely examples




Kirkpatrick Level   Objectives                                            Measures

Reaction            The reaction to the learning material, environment,   Using a Likert type scale participants
                    perception of preparation, facilitator style and      rate the following on average no less
                                                                          than “agree”:
                                                                          a. Preparation
                                                                          b. Facilitator style
                                                                          c. Learning environment
                                                                          d. Participation
                                                                          e. General expectations met
Step 2a: Planning the Intervention (With ROI in Mind)

          What have we achieved by planning the intervention this way?

 ROI ?




Results




                                                                         A path – trail linking the
                                                                         business need with the
                                                                              intervention
Results



Results



Results
Step 2a: Planning the Intervention (With ROI in Mind)

 We can also go some steps further and plan where we will get the information to evaluate
                               the intervention and how…


Kirkpatrick Level   Objectives            Measures       Where Will We Find          In What Format
                                                         Information For
                                                         Evaluation
Impact              See previous slides   See previous   HR Budget vis Actual        Business Report
                                          slides         Staff Turn Over Records
                                                                                     HR Report
Transfer            See previous slides   See previous   Audit/Observation results   Possibly checklists
                                          slides
Learning            See previous slides   See previous   Test scores                 Tests papers and
                                          slides         Practical simulation        answer sheets
                                                         scores                      Simulation checklists
                                                                                     and grading papers
Reaction            See previous slides   See previous   Survey results              Possibly tabulated
                                          slides                                     and presented in
                                                                                     graphs
Step 2b

Planning the Intervention (Planning Beyond
       Training – With ROI In Mind)
Step 2b: Planning the Intervention (With ROI in Mind)

Usually once we have done the planning and developed the training intervention we are
                                ready to implement…




                                                 b. We need to determine what
                                                 technique(s) we will use to show
                                                 that the intervention actually had
                                                 an impact




                              a. With ROI in mind our planning
                              needs to go some steps further…
Step 2b: Planning the Intervention (With ROI in Mind)

We need to determine what technique(s) we will use to show that the intervention actually
                      had an impact – but our dilemma is this…

                                    External Factors
                                     Management
                                      Attention                    OR
                                                                SOMETHING
  AND YES THERE IS                     Incentives                 ELSE?
   IMPROVEMENT
                                  Systems/Procedures
       AFTER
                                       Changes
     PROGRAM

                                  Here is our training            IS IT THE
                                       program                   TRAINING
                                                                 PROGRAM?
Step 2b: Planning the Intervention (With ROI in Mind)

                  We have to isolate the effects of our intervention!




I. Isolating the results/effect of our intervention is the most difficult and also most
   important element of building the case for value (ROI);

II. Because of this, value at the higher levels is seldom determined;

III. However, the credibility not only of your study to show value, but also the training
   function is based on this element.
Step 2b: Planning the Intervention (With ROI in Mind)

                   We have to isolate the effects of our intervention!




I. The complexity of isolating the effect, is a function of the variables that are in play
   at any moment in time in an organization;

II. The analyst should however always try to prove a causal relationship to an
   acceptable degree of accuracy;

III. Acknowledging the forces that influence individual, group and business
   performance is the first step in this process.
Step 2b: Planning the Intervention (With ROI in Mind)

                  We have to isolate the effects of our intervention!




Remember that we are trying to isolate the effect of out intervention to an acceptable
degree of accuracy – this makes very important to involve and agree with the client:

a. What metrics should change as result of the intervention (see step 1)

b. What isolation technique will be used (because they differ in terms of accuracy)




                 Let’s consider the isolation techniques…
Step 2b: Planning the Intervention (With ROI in Mind)

                         Let's look at the techniques!


                                            a. Trend line analysis of performance
                                               data;
    With these                              b. Control groups;
    techniques we
    try to isolate the                      c. Participant’s estimation of impact;
    effects of our
    intervention                            d. Supervisor’s estimation of impact;
                                            e. Management’s estimation of impact;
                                            f. Use of previous studies
                                            g. Subordinate’s report of other factors;
                                            h. Estimating the impact of other factors;
                                            i.   Use of customer input.
Step 2b: Planning the Intervention (With ROI in Mind)

                          Let's look at the techniques!


                                             a. Trend line analysis of performance
                                                data;
                                             b. Control groups;
                                             c. Participant’s estimation of impact;
                                             d. Supervisor’s estimation of impact;

     With these
                                             e. Management’s estimation of impact;
     techniques we
     try to isolate the
                                             f.   Use of previous studies
     effects of other
     interventions
                                             g. Subordinate’s report of other factors;
     and then declare
     the rest as due
                                             h. Estimating the impact of other factors;
     to our effort
                                             i.   Use of customer input.
Step 2b: Planning the Intervention (With ROI in Mind)

                                 Trend Line Analysis




I. Draw a trend-line using previous
   performance data as a base;
II. When intervention is conducted,
    actual performance is plotted and
    compared to the trend line;
III. Improvement in actual performance
     over what the trend line predicted
     can then be reasonably attributed to
     the intervention.
Step 2b: Planning the Intervention (With ROI in Mind)

                   Trend Line Analysis- Advantages and Disadvantages
                  Advantages                                    Disadvantages

I.    Show quick results                       I.    Uses only one factor to isolate the effect
                                                     of the intervention (time- before and
II.   Easy to calculate
                                                     after implementation)
III. Easy to understand
                                               II.   Thus less accurate and client could be
IV. Inexpensive
                                                     sceptical to accept only an analysis of
V. Not difficult to organize (basically just         the trend line.
      monitor changes in metrics)

VI. A good first order analysis
Step 2b: Planning the Intervention (With ROI in Mind)

                      Example- Not from our central case study…




                                                             Effect of the
                                                             intervention
COMPLAINTS




                              Average
                          Pre Program 55

                                                   Average
                          Sexual
                                               Post Program 35
                        Harassment
                    Prevention Program

               O N D J F M A M J J A S O N D J F M A M J J
               A S O               TIME
Step 2b: Planning the Intervention (With ROI in Mind)

                                  Control Groups Technique



I.   Similar to classical research the control group method, involves an experimental group and a
     control group;
II. The so called experimental group will be the target of our intervention;
III. The control group would be a group that is paired with the experimental group, but would
     not undergo the intervention;
IV. Important things to remember with this method are:
      a.    Paring need to be thought about carefully so as not to dilute the impact of the
            intervention;
      b.    Although not as rigid as classical research key factors that could dilute the impact or
            assessment of the impact of the intervention need to be controlled.
Step 2b: Planning the Intervention (With ROI in Mind)

                         Control Groups- Advantages and Disadvantages
                     Advantages                                          Disadvantages

I.    More accurate than most techniques             I.    More complicated

II.   If the pairing and control of variables are    II.   Require more planning, negotiation,
      agreed with the client the outcome has high          control and monitoring
      level of credibility
                                                     III. As result of above many practitioners are
III. With this control groups, given the level of          unwilling to use this technique
      accuracy it is possible to benchmark the ROI
                                                     IV. Requires absolute agreement with the
      of certain interventions given a
                                                           client environment that variables will be
      predetermined set of variables
                                                           kept constant for the duration of the ROI
                                                           evaluation.
Step 2b: Planning the Intervention (With ROI in Mind)

                 Schematic Explanation of control groups design

1. Pairing
                                                            2.
                                                         Control
                                                        Variables


             Control
                                    M1                                        M2
             Group



             Experimental
                                    M1             Intervention               M2
             Group

                                                                    *M = measurement
Step 2b: Planning the Intervention (With ROI in Mind)

                  Looking at pairing and control of variables


                 PAIRING
                                              Note:
                                              I. Too small then meaningful pairing is not
                                                   possible – too big then you are not going
                                                   to be able to control the variables during
                                                   and after the implementation of your
         Select your                               intervention;
experimental and control group                II. Remember we go for reasonable
          carefully                                accuracy therefore you pair the major
                                                   elements and also control the major
                                                   variables!!
                                              III. The above makes it very important to
                                                   agree upfront with your client
                                                   environment what how pairing will be
  VARIABLES                                        done and which variables will be
                                                   controlled.
Step 2b: Planning the Intervention (With ROI in Mind)

               So what do you typically pair?


                      Pair for (Primarily)




                                                                               General demography
                                                           Performance level
                                        Leadership stype
                       Nature of work
               Size




                      Negotiate creafully
Step 2b: Planning the Intervention (With ROI in Mind)

                                             So what do you typically try to control?


                      Changes in:                                                                             Changes in:




                                                                                                                                                       Systems and procedures
                                                                                                                  Recognition and reward
                                                                 General demography
                                             Performance level
                          Leadership stype
         Nature of work




                                                                                                                                           Turn over
                                                                                          structure

                                                                                                       Function
                                                                                      +
Size




        Negotiate creafully                                                                           Negotiate creafully
Step 2b: Planning the Intervention (With ROI in Mind)

   Remember your decision about pairing and control of variables….




        = (f) (your knowledge and
      judgement) + agreement with
         the client environment +
                practicability
Step 2b: Planning the Intervention (With ROI in Mind)
  Estimation of impact by third parties ( this technique can include an estimation of the
perceived impact of the intervention, by participants, participants supervisors, participants
                                        managers)


    This technique provides participants with a questionnaire, and based on the
    improvement ask them to respond to the following:
    I. What percent of this improvement can be attributed to the application of
       skills/techniques/knowledge gained in the training program?
    II. What is the basis for this estimation?
    III. What confidence do you have in this estimate, expressed as a percentage?
    IV. What other factors contributed to this improvement in performance?
    V. What other individuals or groups could estimate this percentage or determine
       the amount?
Step 2b: Planning the Intervention (With ROI in Mind)

Estimation of impact by third parties – how do you increase the confidence level for this
                                       technique?



  I.   Individuals who do not respond to the questionnaire or provide non- usable data
       on the questionnaire are assumed to report no improvement/impact;
  II. Extreme data and unrealistic claims are omitted from the analysis;
  III. Only annualized values are used, it is assumed that there are no benefits from
       the program after the first year of implementation;
  IV. The confidence level, expressed as a percent, is multiplied by the improvement
      value to reduce the amount of the improvement by the potential error;
  V. If value is expressed it is also factored by the confidence value.
Step 2b: Planning the Intervention (With ROI in Mind)

        Estimation Of Impact By Third Parties - Advantages And Disadvantages
                   Advantages                                        Disadvantages

I.    Easy to implement;                             I.    Less accurate than control groups
II.   Based on the assumption that participants      II.   Based on perceptions – could be
      are capable of determining or estimating
                                                           regarded as Return on Expectation
      impact;
                                                           rather than ROI
III. Although an estimate, this value will usually
                                                     III. Could become victim of “group think”
      have considerable credibility with
      management because participants are at the
      centre of the change or improvement;

IV. High face validity.
Step 2b: Planning the Intervention (With ROI in Mind)

     Example from an actual coaching intervention- Not from our central case study…

                                                                             Participants
                                                                             and
                                                                             management
                                                                             agreed which
All indicators                                                               the respective
were pre                                                                     groups will
agreed with the                                                              rate
client
Step 2b: Planning the Intervention (With ROI in Mind)

                             So now we can complete our plan
Kirkpatrick   Objectives      Measures   Where Will We       In What          When              Purpose of
Level                                    Find Information    Format                             Data
                                         For Evaluation                                         Collection and
                                                                                                Analysis
Impact        See previous    See        HR Budget vis       Business         Six months        ROI and
              slides          previous   Actual              Report           after final       Training
                              slides     Staff Turn Over                      course was        Evaluation
                                         Records             HR Report        presented
Transfer      See previous    See        Audit/Observatio    Possibly         Period of three   Training
              slides          previous   n results           checklists       months            Evaluation
                              slides                                          following
                                                                              training
Learning      See previous    See        Test scores         Tests papers     Within one        Training
              slides          previous   Practical           and answer       week following    Evaluation
                              slides     simulation scores   sheets           training
                                                             Simulation
                                                             checklists and
                                                             grading papers
Reaction      See previous    See        Survey results      Possibly         Within one        Training
              slides          previous                       tabulated and    week following    Evaluation
                              slides                         presented in     training
                                                             graphs
Step 2b: Planning the Intervention (With ROI in Mind)
Note – because the techniques related to isolating the impact of other factors are
                not used often it is only referred to in this toolkit…

                                               a. Trend line analysis of performance
                                                  data;
                                               b. Control groups;
                                               c. Participant’s estimation of impact;
                                               d. Supervisor’s estimation of impact;
              With these                       e. Management’s estimation of impact;
              techniques we
              try to isolate the               f.   Use of previous studies
              effects of other
              interventions                    g. Subordinate’s report of other factors;
              and then declare
              the rest as due                  h. Estimating the impact of other factors;
              to our effort
                                               i.   Use of customer input.
Step 3

Implementation and Data Gathering (With ROI in
                   Mind)
Implementation and Data Gathering (With ROI in Mind)

Referring back to our original example (turnover in graduate engineers) – let us assume the
                                         following:



                             Correct                                          Correct
                             pairing                                          pairing

                             Correct                                          Correct
                             control of                                       control of
                             variables                                        variables

                             Agreement                                        Agreement
      Mine A                 on metrics
                                                          Mine B              on metrics
      Control                                          Experiential


                                 We opted for a control group
                                     evaluation method
Implementation and Data Gathering (With ROI in Mind)

Referring back to our original example (turnover in graduate engineers) – let us assume the
                                         following:




                                         Impact


                                        Transfer
                                 -Evaluation completed                Six months after the
                                                                        intervention the
                                                                            need for
                             Learning -Evaluation completed              replacement of
                                                                         graduates were
                                                                      down by 50% in the
                                                                          experimental
                                  Reaction – Evaluation                       group
                                       completed
Implementation and Data Gathering (With ROI in Mind)

Referring back to our original example (turnover in graduate engineers) – let us assume the
                                         following:




                                         Impact


                                        Transfer
                                 -Evaluation completed
                                                                      In the control group the
                                                                      need for replacement of
                                                                      graduate engineers also
                             Learning -Evaluation completed           down by 10%...possibly
                                                                      because of anticipation
                                                                          of changes in HR
                                                                              approach
                                  Reaction – Evaluation
                                       completed
Implementation and Data Gathering (With ROI in Mind)

Referring back to our original example (turnover in graduate engineers) – let us assume the
                                         following:




                                         Impact


                                        Transfer
                                 -Evaluation completed
                                                                      In the control group the
                                                                      need for replacement of
                                                                      graduate engineers also
                             Learning -Evaluation completed           down by 10%...possibly
                                                                      because of anticipation
                                                                          of changes in HR
                                                                              approach
                                  Reaction – Evaluation
                                       completed
So because to the method we used we can assign the
change in te metric to our intervention, mainly because we
    controlled major variables. We do this as follows…
Step 4

 Converting the Change As Result of the
Intervention to Value and Calculating ROI
Step 4.a Converting the Change As Result of the
                        Intervention to Value
Referring back to our original example (turnover in graduate engineers)…this is how it is
                                         dome…

        Identify the unit of    In our case study:        Value per unit =
        improvement             Number of                 R 74964 (see slide 12
                                replacements of           AND 13
                                graduate engineers in
                                the past six months
        Determine the           Original replacements =   Original cost of
        performance level       6                         replacement
        change                  Six months after the      R 449784
                                intervention = 3          Our latest cost of
                                Improvement of 50%        replacement following
                                                          intervention
                                                          R 224892

        Calculate the           50% saving                R 224892
        performance
        improvement
Step 4.a Converting the Change As Result of the
                           Intervention to Value
Because this is such a vast area to know here is some additional input re calculating the value
                                     of change in a metric..




                              Primary Measurements
                                 of Improvement
Step 4.a Converting the Change As Result of the
                 Intervention to Value



     Always try to link to a metric –
    alternatively use the method of
Estimation Of Impact By Third Parties –
See slide 55 and 56 (this method is also
 referred to as Return on Expectation)
Step 4.a Converting the Change As Result of the
                            Intervention to Value
                            See whether one can link tangible values to these?
OUTPUT                                               TIME
Units Produced                                       Equipment Downtime
Tons Manufactured                                    Overtime
Items Assembled                                      Time to Project Completion
Money Collected                                      Processing Time
Items Sold                                           Break in Time for New Employees
Forms Processed                                      Learning Time
Loans Approved                                       Meeting Schedules
Inventory Turnover                                   Repair Time
Patients Visited                                     Work Stoppages
Applications Processed                               Order Response
Tasks Completed                                      Late Reporting
Output Per Hour                                      Lost Time Days
Productivity                                         QUALITY
New Accounts Generated                               Scrap
COSTS                                                Waste
Budget Variances                                     Rejects
Unit Costs                                           Error Rates
Cost By Account                                      Rework
Variable Costs                                       Shortages
Fixed Costs                                          Product Defects
Overhead Cost                                        Deviation From Standard
Operating Costs                                      Product Failures
Number of Cost Reductions                            Inventory Adjustments
Project Cost Savings                                 Time Card Corrections
Accident Costs                                       Percent of Tasks Completed Properly
Sales Expense                                        Number of Accidents
Step 4.a Converting the Change As Result of the
                                  Intervention to Value
                                        And these….
WORK HABITS                                   CUSTOMER SERVICE

Absenteeism                                   Customer Complaints
Tardiness                                     Customer Satisfaction
Visits to the Dispensary                      Customer Dissatisfaction
First Aid Treatments                          Customer Impressions
Violations of Safety Rules                    Customer Loyalty
Number of Communication Break-downs           Customer Retention
Excessive Breaks                              Customer Value
Follow-Up                                     Lost Customers
                                              EMPLOYEE DEVELOPMENT/ADVANCEMENT
WORK CLIMATE/SATISFACTION                     Number of Promotions
                                              Number of Pay Increases
Number of Grievances                          Number of Learning Programs Attended
Number of Discrimination Charges              Requests for Transfer
Employee Complaints                           Performance Appraisal Ratings
Job Satisfaction                              Increases in Job Effectiveness
Employee Turnover
Litigation                                    INITIATIVE/INNOVATION
Organization Commitment                       Implementation of New Ideas
Employee Loyalty                              Successful Completion of Projects
Increased Confidence                          Number of Suggestions Implemented
                                              Setting Goals and Objectives
                                              New Products and Services Developed
                                              New Patents and Copyrights
Step 4.a Converting the Change As Result of the
                    Intervention to Value
         Or determining value on a case by case basis using historical data


                                             Legal fee;
                  Actual cost
                                            Settlement
                 from records
                                                cost

   Total of                                                        Cost
35 complaints                                                   R 285,000


                   Estimated               Staff time;
                   additional             Management
                   from staff                 time

                 R 285,000/35 = Cost per complaint
                Cost of Sexual Harassment Complaint
Step 4.a Converting the Change As Result of the
                  Intervention to Value




In our case example we did determine the value of
  the change and can therefore now move on to
   calculate cost benefit and therefore also ROI
Step 4.b Calculating ROI
    First thing to do now is to determine the cost of the intervention – again back to our main
                                             example …

Cost element                               Notes                                           ZAR value (for the sake of our example)

Development Costs                          If you use more than one trainer or one         R50,000
                                           trainer on more than one project – pro-rate
Program Materials                                                                          R10,000

Instructor/Facilitator Costs               If you use more than one trainer or one         R10,000
                                           trainer on more than one project – pro-rate
Facilities Costs                                                                           R10,000

Travel and lodging                                                                         R10,000

Salaries and time loss of participants     Will need to consult HR and line and pro-rate   R50,000

Administration costs                                                                       R5,000

Evaluation costs                                                                           R 5,000

Total cost of the intervention                                                             R150,000
Step 4.b Calculating ROI
First thing to do now is to determine the cost of the intervention – again back to our main
                                         example …



                                          Net Program Benefits (R224892- R150,000)
    Benefit/Cost Ratio           =
                                                  Program Costs (R150,000)




                                     Ratio = 0.499928
Step 4.b Calculating ROI
First thing to do now is to determine the cost of the intervention – again back to our main
                                         example …



                       Net Program Benefits (R224892- R150,000)        X 100
     Benefits ROI =
                               Program Costs (R150,000)                   1




                                  Benefit ROI = 50% thus for
                                    every R1 I spent on this
                                 program you got your money
                                  back through saving as well
                                   as an additional 50 cents.

Mais conteúdo relacionado

Semelhante a ROI on People Management Interventions - It is Possible!!

MGT 3302, Introduction to Project Management 1 Cou.docx
MGT 3302, Introduction to Project Management 1 Cou.docxMGT 3302, Introduction to Project Management 1 Cou.docx
MGT 3302, Introduction to Project Management 1 Cou.docxmadlynplamondon
 
1 OL 325 Final Project Case Study Designing a Pay S
1  OL 325 Final Project Case Study  Designing a Pay S1  OL 325 Final Project Case Study  Designing a Pay S
1 OL 325 Final Project Case Study Designing a Pay SVannaJoy20
 
Mb0041 mit set 2
Mb0041 mit set 2Mb0041 mit set 2
Mb0041 mit set 2Sampath Raj
 
Chapter11 economics
Chapter11 economicsChapter11 economics
Chapter11 economicsVin Voro
 
Operating costs for small businesses new format
Operating costs for small businesses   new formatOperating costs for small businesses   new format
Operating costs for small businesses new formatAroostookComputer
 
Taguchi method
Taguchi methodTaguchi method
Taguchi methodyaka9217
 
Ac315 chapter4lecturenotes
Ac315 chapter4lecturenotesAc315 chapter4lecturenotes
Ac315 chapter4lecturenotesknappr3
 
Assignment help & writing Services By Global Assignment Help
Assignment help & writing Services By Global Assignment HelpAssignment help & writing Services By Global Assignment Help
Assignment help & writing Services By Global Assignment HelpAmelia Jones
 
SDPM - Lecture 2a - Project evaluation – for the buyer, and for the vendor
SDPM - Lecture 2a - Project evaluation – for the buyer, and for the vendorSDPM - Lecture 2a - Project evaluation – for the buyer, and for the vendor
SDPM - Lecture 2a - Project evaluation – for the buyer, and for the vendorOpenLearningLab
 
Selected solutions 1
Selected solutions   1Selected solutions   1
Selected solutions 1Tlepoorit
 
Handout 3 entrepreneurship - call of the day (1)
Handout 3   entrepreneurship - call of the day (1)Handout 3   entrepreneurship - call of the day (1)
Handout 3 entrepreneurship - call of the day (1)Prashanth Karthikeyan
 
Calculating the ROI of collaboration tools
Calculating the ROI of collaboration tools Calculating the ROI of collaboration tools
Calculating the ROI of collaboration tools K COMPANY
 

Semelhante a ROI on People Management Interventions - It is Possible!! (20)

MGT 3302, Introduction to Project Management 1 Cou.docx
MGT 3302, Introduction to Project Management 1 Cou.docxMGT 3302, Introduction to Project Management 1 Cou.docx
MGT 3302, Introduction to Project Management 1 Cou.docx
 
1 OL 325 Final Project Case Study Designing a Pay S
1  OL 325 Final Project Case Study  Designing a Pay S1  OL 325 Final Project Case Study  Designing a Pay S
1 OL 325 Final Project Case Study Designing a Pay S
 
Mb0041 mit set 2
Mb0041 mit set 2Mb0041 mit set 2
Mb0041 mit set 2
 
Chapter11 economics
Chapter11 economicsChapter11 economics
Chapter11 economics
 
Operating costs for small businesses new format
Operating costs for small businesses   new formatOperating costs for small businesses   new format
Operating costs for small businesses new format
 
Taguchi method
Taguchi methodTaguchi method
Taguchi method
 
Assign9
Assign9Assign9
Assign9
 
Technical Debt
Technical DebtTechnical Debt
Technical Debt
 
Costs (3).pptx
Costs (3).pptxCosts (3).pptx
Costs (3).pptx
 
Ac315 chapter4lecturenotes
Ac315 chapter4lecturenotesAc315 chapter4lecturenotes
Ac315 chapter4lecturenotes
 
Assignment help & writing Services By Global Assignment Help
Assignment help & writing Services By Global Assignment HelpAssignment help & writing Services By Global Assignment Help
Assignment help & writing Services By Global Assignment Help
 
Qc story
Qc storyQc story
Qc story
 
8d_en.pdf
8d_en.pdf8d_en.pdf
8d_en.pdf
 
Strategic cost management
Strategic cost managementStrategic cost management
Strategic cost management
 
SDPM - Lecture 2a - Project evaluation – for the buyer, and for the vendor
SDPM - Lecture 2a - Project evaluation – for the buyer, and for the vendorSDPM - Lecture 2a - Project evaluation – for the buyer, and for the vendor
SDPM - Lecture 2a - Project evaluation – for the buyer, and for the vendor
 
Selected solutions 1
Selected solutions   1Selected solutions   1
Selected solutions 1
 
Handout 3 entrepreneurship - call of the day (1)
Handout 3   entrepreneurship - call of the day (1)Handout 3   entrepreneurship - call of the day (1)
Handout 3 entrepreneurship - call of the day (1)
 
Technical debt
Technical debtTechnical debt
Technical debt
 
Calculate overhead
Calculate overheadCalculate overhead
Calculate overhead
 
Calculating the ROI of collaboration tools
Calculating the ROI of collaboration tools Calculating the ROI of collaboration tools
Calculating the ROI of collaboration tools
 

Mais de dmdk12

The Case for Change
The Case for ChangeThe Case for Change
The Case for Changedmdk12
 
Framework for Project Layout Meeting
Framework for Project Layout MeetingFramework for Project Layout Meeting
Framework for Project Layout Meetingdmdk12
 
Change Presented ad A Project Roadmap: Infographic Template
Change Presented ad A Project Roadmap: Infographic TemplateChange Presented ad A Project Roadmap: Infographic Template
Change Presented ad A Project Roadmap: Infographic Templatedmdk12
 
The Change Project As an Archetypal Journey
The Change Project As an Archetypal JourneyThe Change Project As an Archetypal Journey
The Change Project As an Archetypal Journeydmdk12
 
Change Journey as an Individual Quest
Change Journey as an Individual Quest Change Journey as an Individual Quest
Change Journey as an Individual Quest dmdk12
 
Infographic Template for Developing a Project Journey
Infographic Template for Developing a Project JourneyInfographic Template for Developing a Project Journey
Infographic Template for Developing a Project Journeydmdk12
 
The People Management Story
The People Management StoryThe People Management Story
The People Management Storydmdk12
 
Infographic - Steering Committee Purpose, Role, Requirements, Metrics
Infographic - Steering Committee Purpose, Role, Requirements, MetricsInfographic - Steering Committee Purpose, Role, Requirements, Metrics
Infographic - Steering Committee Purpose, Role, Requirements, Metricsdmdk12
 
The HR Career in South Africa
The HR Career in South AfricaThe HR Career in South Africa
The HR Career in South Africadmdk12
 
LEadership Theories Snap Shot
LEadership Theories Snap ShotLEadership Theories Snap Shot
LEadership Theories Snap Shotdmdk12
 
The People Management Story
The People Management StoryThe People Management Story
The People Management Storydmdk12
 
Model for Merger Management - This is a work in progress!!
Model for Merger Management - This is a work in progress!!Model for Merger Management - This is a work in progress!!
Model for Merger Management - This is a work in progress!!dmdk12
 
Coaching Toolkit
Coaching ToolkitCoaching Toolkit
Coaching Toolkitdmdk12
 
Focus on CPM and Resp Matrix
Focus on CPM and Resp MatrixFocus on CPM and Resp Matrix
Focus on CPM and Resp Matrixdmdk12
 
Iron Triangle
Iron TriangleIron Triangle
Iron Triangledmdk12
 
The Project Cycle Revisited
The Project Cycle RevisitedThe Project Cycle Revisited
The Project Cycle Revisiteddmdk12
 
Projects- The Three Artifacts
Projects- The Three ArtifactsProjects- The Three Artifacts
Projects- The Three Artifactsdmdk12
 
Organization Design 2 - Focus on Strategy
Organization Design 2 - Focus on StrategyOrganization Design 2 - Focus on Strategy
Organization Design 2 - Focus on Strategydmdk12
 
The One Slide Change Mgt Model
The One Slide Change Mgt ModelThe One Slide Change Mgt Model
The One Slide Change Mgt Modeldmdk12
 
Organization Design 1
Organization Design 1Organization Design 1
Organization Design 1dmdk12
 

Mais de dmdk12 (20)

The Case for Change
The Case for ChangeThe Case for Change
The Case for Change
 
Framework for Project Layout Meeting
Framework for Project Layout MeetingFramework for Project Layout Meeting
Framework for Project Layout Meeting
 
Change Presented ad A Project Roadmap: Infographic Template
Change Presented ad A Project Roadmap: Infographic TemplateChange Presented ad A Project Roadmap: Infographic Template
Change Presented ad A Project Roadmap: Infographic Template
 
The Change Project As an Archetypal Journey
The Change Project As an Archetypal JourneyThe Change Project As an Archetypal Journey
The Change Project As an Archetypal Journey
 
Change Journey as an Individual Quest
Change Journey as an Individual Quest Change Journey as an Individual Quest
Change Journey as an Individual Quest
 
Infographic Template for Developing a Project Journey
Infographic Template for Developing a Project JourneyInfographic Template for Developing a Project Journey
Infographic Template for Developing a Project Journey
 
The People Management Story
The People Management StoryThe People Management Story
The People Management Story
 
Infographic - Steering Committee Purpose, Role, Requirements, Metrics
Infographic - Steering Committee Purpose, Role, Requirements, MetricsInfographic - Steering Committee Purpose, Role, Requirements, Metrics
Infographic - Steering Committee Purpose, Role, Requirements, Metrics
 
The HR Career in South Africa
The HR Career in South AfricaThe HR Career in South Africa
The HR Career in South Africa
 
LEadership Theories Snap Shot
LEadership Theories Snap ShotLEadership Theories Snap Shot
LEadership Theories Snap Shot
 
The People Management Story
The People Management StoryThe People Management Story
The People Management Story
 
Model for Merger Management - This is a work in progress!!
Model for Merger Management - This is a work in progress!!Model for Merger Management - This is a work in progress!!
Model for Merger Management - This is a work in progress!!
 
Coaching Toolkit
Coaching ToolkitCoaching Toolkit
Coaching Toolkit
 
Focus on CPM and Resp Matrix
Focus on CPM and Resp MatrixFocus on CPM and Resp Matrix
Focus on CPM and Resp Matrix
 
Iron Triangle
Iron TriangleIron Triangle
Iron Triangle
 
The Project Cycle Revisited
The Project Cycle RevisitedThe Project Cycle Revisited
The Project Cycle Revisited
 
Projects- The Three Artifacts
Projects- The Three ArtifactsProjects- The Three Artifacts
Projects- The Three Artifacts
 
Organization Design 2 - Focus on Strategy
Organization Design 2 - Focus on StrategyOrganization Design 2 - Focus on Strategy
Organization Design 2 - Focus on Strategy
 
The One Slide Change Mgt Model
The One Slide Change Mgt ModelThe One Slide Change Mgt Model
The One Slide Change Mgt Model
 
Organization Design 1
Organization Design 1Organization Design 1
Organization Design 1
 

ROI on People Management Interventions - It is Possible!!

  • 1. ROI of Human Capital Interventions A Practice Toolkit
  • 2. Description Icon Relevant quote Concept/Framework Tools, Techniques Template
  • 3. Step 1 Presenting Problem to Real Problem
  • 4. Step 1. Presenting Problem to Real Problem The first task of the practitioner is almost always analysis Starts with a “presenting” problem – not a “hunch” but a problem … as it presents itself
  • 5. Step 1. Presenting Problem to Real Problem Example… Let's take an example – higher staff turnover for Graduate Engineers (let say compared to an industry standard) The first question should not be, what skills and knowledge are required. It should be which metric(s) are affected negatively.
  • 6. Step 1. Presenting Problem to Real Problem Example… Cost to fill a vacant On boarding & position Orientation cost Cost of 'covering' a vacant position (Calculated costs of Cost of productivity other employees ramp-up 'filling in' while the position is vacant) Unit cost per replacement
  • 7. Step 1. Presenting Problem to Real Problem 1.1 Calculating benchmark employee cost Nr Metric Element Description Example 1. Departing employee annual base N/a 100,000 salary 2. Calculated annual benefits cost Estimated at 20% of 20,000 base salary 3. Calculated monthly salary + benefits 10,000 4. Calculated daily salary + benefits Based on 230 (8hr) 522 working days
  • 8. Step 1. Presenting Problem to Real Problem 1.2 Cost of 'covering' a vacant position (Calculated costs of other employees 'filling in' while the position is vacant) Nr Metric Element Description Example 1. Number of days until the vacant N/a 90 position is filled 2. Calculated daily cost of 'covering' a 33% of departing 522/3 vacant position employee's daily salary + benefits 3. Total cost to 'cover' vacant position 15644
  • 9. Step 1. Presenting Problem to Real Problem 1.3 Cost to fill a vacant position Nr Metric Element Description Example 1. HR/Hiring manager's annual salary N/a 200,000 2. Calculated HR/Hiring manager's hourly rate Based on 230 (8hr) working 130 days and 20% fringe rate 3. Cost of advertising (online and/or print) Enter cost 3000 4. Cost of resume screening HR/Hire manager hours * 50*130=6500 hourly rate 5. Cost of interviews (telephone screening, 1st and HR/Hire manager hours * 50*130=6500 2nd) hourly rate 6. Cost of behavioural and skills assessments Enter cost 3000 7. Cost of background checks (criminal, credit, Enter cost 4000 reference, education) 8. Cost of travel/moving expenses (if applicable) Enter cost 5000 9. Total cost to fill a vacant position: 28,000
  • 10. Step 1. Presenting Problem to Real Problem 1.4 On boarding & Orientation cost Nr Metric Element Description Example 1. Trainer/Manager annual salary: N/a 100,000 2. Calculated trainer/manager daily rate Based on 230 (8hr) working 522 days and 20% fringe rate 3. Total training days Enter number of days 10 4. Total on boarding and orientation cost N/a 5220
  • 11. Step 1. Presenting Problem to Real Problem 1.5 Cost of productivity ramp-up (During the first 3 months, an average new employee performs at 50% productivity of a tenured top performing employ Nr Metric Element Description Example 1. Daily employee cost (salary + benefits) N/a 522 2. Number of working days during first 3 months: Enter number of days (avg. 50 58 days 3. Cost of productivity ramp-up N/a 26100
  • 12. Step 1. Presenting Problem to Real Problem 1.6 Unit cost for filling a vacancy R 74964 Two important learning points: I. Know your metrics II. Negotiate, and even request help from your client to determine the right metrics. They are closer to the business and at the same time you obtain by in for the intervention.
  • 13. Step 1. Presenting Problem to Real Problem 1.7 Total cost of replacing graduate engineers R 74964 x 6 = R 449784 Let's assume that we had to replace five graduate engineers the past six months
  • 14. Step 1. Presenting Problem to Real Problem Now we can move on to analysis We know now which metrics are affected Consultant speciality Presenting Analysis and Identification of Solution Intervention Problem/ Definition of Performance Competence Gap Generation Strategy Opportunity Business Need Need Non Training Solution
  • 15. Step 1. Presenting Problem to Real Problem The basic tasks in analysis 1. Data gathering 2. Data consolidation 3. Data analysis 4. Perspective and information for decision making
  • 16. Step 1. Presenting Problem to Real Problem Two types of data Quantitative ”The truth is in a number” – however does not say much about context Data ” More about context, how people experience in their own words”. Qualitative
  • 17. Step 1. Presenting Problem to Real Problem Data gathering I. Questionnaires II. Surveys III. Tests IV. Interviews V. Focus groups VI. Observation VII. Performance records VIII.Knowledge and skills testing IX. Program follow up X. Project assignments
  • 18. Step 1. Presenting Problem to Real Problem Data gathering I. Target population/participants; II. Supervisors of target population/participants; III. Subordinates of target population/participants; IV. Peer Group; V. HRD Staff; VI. Documentation: • Organizational Performance Records: • Disciplinary records; • Safety records; • Scorecards
  • 19. Step 1. Presenting Problem to Real Problem Contextual factors to consider when selecting data collection methods I. Time for participants to respond; II. Costs; III.Amount of disruption of normal activities; IV. What is the level of certainty that you will achieve with the result; V. Practicality; VI. Culture/Philosophy – example the 10% syndrome.
  • 20. Step 1. Presenting Problem to Real Problem Requirements for effective data collection instruments and techniques I. Construct validity ( methods, techniques and test instruments should be evaluated in terms of its theoretical grounding); II. Predictive validity (methods, techniques and test instruments should be evaluated regularly in terms of its ability to predict what it purport to predict); III. Face validity – for line management very important!
  • 21. Step 1. Presenting Problem to Real Problem Requirements for effective data collection instruments and techniques Reliability: I. Produce consistent results over time, that differentiates effectively, and are independent of the assessor/data collector, or context.
  • 22. Step 1. Presenting Problem to Real Problem Data consolidation Data Consolidation Techniques are Wide and Varied Descriptive statistics Qualitative description, using organizational and other models and concepts
  • 23. Step 1. Presenting Problem to Real Problem Data analysis Data Analysis Varies According to Situation Descriptive and inferential statistics Qualitative decisions about the situation, based on decision making and techniques
  • 24. Note the following ! • Because this toolkit does not aim to transfer knowledge about analysis as such the previous slides provides a brief and high level overview, for the sake of a clearity… for the sake of the toolkit we assume that the analysis provided clarity about the problem….
  • 25. Step 1. Presenting Problem to Real Problem Brief Exposition of Problem Our analysis shows that the HR and line managers involved in selection decisions do interviews in a haphazard and nonscientific Lets assume fashion this based on the core Solution example Following discussions with consultants, HR experts and line all agreed we need to implement competency based interviewing
  • 26. Step 1. Presenting Problem to Real Problem The training department is instructed to develop an appropriate program and put all line and support staff through training – they however indicate that a change in results is required quickly and that ROI for the training need to be justified soon after implementation… Lets assume this based on the core example
  • 27. Presenting Problem to Real Problem So we have addressed this even before we asked any questions about training – and now we can move on to the next step Starts with a “presenting” problem – not a “hunch” but a problem … as it presents itself
  • 28. Step 2a Planning the Intervention (Starting With the End In Mind)
  • 29. Step 2a: Planning the Intervention (With ROI in Mind) When we plan an intervention – considering that we want to measure ROI- we really reverse engineer (or start with the end in mind) – We use the Kirkpatrick Model to achieve this 4. Impact What is the business impact we need to achieve (refer step 1 where we got a firm grip on the metric that must change) 3. Transfer What is the work place behaviour that must change 2. Learning What knowledge, skills and attitudes need to be acquired 1. Reaction What is the required reaction of the learner to the intervention
  • 30. Step 2a: Planning the Intervention (With ROI in Mind) Back to our case study to show how we plan “backwards” – remember these objectives and measures for clarification, based on our core example/case study (re graduate engineers) Kirkpatrick Level Objectives Measures Impact Train target population (define) in behaviour based 25% reduction in total cost of filling interviewing techniques, with the aim to: vacancies for graduate engineers a. Improve predictive validity of selection decisions b. Driving down cost of filling vacancies for graduate engineers Transfer Correct application of behaviour based interviewing Expert observation with the aid of techniques by target populations during selection checklists show that all involved in interviewing selection interviews use the correct interviewing structure (agenda) and questioning techniques at least 85% of the time. Observation is conducted for two months following the initial training and corrective coaching is implemented where required, following interviews.
  • 31. Step 2a: Planning the Intervention (With ROI in Mind) Back to our case study to show how we plan “backwards” – remember these objectives and measures a merely examples Kirkpatrick Level Objectives Measures Learning Using Blooms Taxonomy of Learning Objectives here are 100% compliance with the Bloom a few examples of the objectives that will be outlined Taxonomy, alignment with SAQA Unit here: Standards and also SAQA guidelines a. The learner should demonstrate that they can and requirements for assessment analyse a CV, with the aim to develop behaviour and moderation; based questions for an interview; Learners score 75% on theoretical b. The learner should demonstrate the ability to paper to pass and 80% on practical develop interview questions according to the simulation interview. behavioural interview format c. Etc. etc… Remember this is a snapshot – there is a whole learning map and course development process underlying these objectives
  • 32. Step 2a: Planning the Intervention (With ROI in Mind) Back to our case study to show how we plan “backwards” – remember these objectives and measures a merely examples Kirkpatrick Level Objectives Measures Reaction The reaction to the learning material, environment, Using a Likert type scale participants perception of preparation, facilitator style and rate the following on average no less than “agree”: a. Preparation b. Facilitator style c. Learning environment d. Participation e. General expectations met
  • 33. Step 2a: Planning the Intervention (With ROI in Mind) What have we achieved by planning the intervention this way? ROI ? Results A path – trail linking the business need with the intervention Results Results Results
  • 34. Step 2a: Planning the Intervention (With ROI in Mind) We can also go some steps further and plan where we will get the information to evaluate the intervention and how… Kirkpatrick Level Objectives Measures Where Will We Find In What Format Information For Evaluation Impact See previous slides See previous HR Budget vis Actual Business Report slides Staff Turn Over Records HR Report Transfer See previous slides See previous Audit/Observation results Possibly checklists slides Learning See previous slides See previous Test scores Tests papers and slides Practical simulation answer sheets scores Simulation checklists and grading papers Reaction See previous slides See previous Survey results Possibly tabulated slides and presented in graphs
  • 35. Step 2b Planning the Intervention (Planning Beyond Training – With ROI In Mind)
  • 36. Step 2b: Planning the Intervention (With ROI in Mind) Usually once we have done the planning and developed the training intervention we are ready to implement… b. We need to determine what technique(s) we will use to show that the intervention actually had an impact a. With ROI in mind our planning needs to go some steps further…
  • 37. Step 2b: Planning the Intervention (With ROI in Mind) We need to determine what technique(s) we will use to show that the intervention actually had an impact – but our dilemma is this… External Factors Management Attention OR SOMETHING AND YES THERE IS Incentives ELSE? IMPROVEMENT Systems/Procedures AFTER Changes PROGRAM Here is our training IS IT THE program TRAINING PROGRAM?
  • 38. Step 2b: Planning the Intervention (With ROI in Mind) We have to isolate the effects of our intervention! I. Isolating the results/effect of our intervention is the most difficult and also most important element of building the case for value (ROI); II. Because of this, value at the higher levels is seldom determined; III. However, the credibility not only of your study to show value, but also the training function is based on this element.
  • 39. Step 2b: Planning the Intervention (With ROI in Mind) We have to isolate the effects of our intervention! I. The complexity of isolating the effect, is a function of the variables that are in play at any moment in time in an organization; II. The analyst should however always try to prove a causal relationship to an acceptable degree of accuracy; III. Acknowledging the forces that influence individual, group and business performance is the first step in this process.
  • 40. Step 2b: Planning the Intervention (With ROI in Mind) We have to isolate the effects of our intervention! Remember that we are trying to isolate the effect of out intervention to an acceptable degree of accuracy – this makes very important to involve and agree with the client: a. What metrics should change as result of the intervention (see step 1) b. What isolation technique will be used (because they differ in terms of accuracy) Let’s consider the isolation techniques…
  • 41. Step 2b: Planning the Intervention (With ROI in Mind) Let's look at the techniques! a. Trend line analysis of performance data; With these b. Control groups; techniques we try to isolate the c. Participant’s estimation of impact; effects of our intervention d. Supervisor’s estimation of impact; e. Management’s estimation of impact; f. Use of previous studies g. Subordinate’s report of other factors; h. Estimating the impact of other factors; i. Use of customer input.
  • 42. Step 2b: Planning the Intervention (With ROI in Mind) Let's look at the techniques! a. Trend line analysis of performance data; b. Control groups; c. Participant’s estimation of impact; d. Supervisor’s estimation of impact; With these e. Management’s estimation of impact; techniques we try to isolate the f. Use of previous studies effects of other interventions g. Subordinate’s report of other factors; and then declare the rest as due h. Estimating the impact of other factors; to our effort i. Use of customer input.
  • 43. Step 2b: Planning the Intervention (With ROI in Mind) Trend Line Analysis I. Draw a trend-line using previous performance data as a base; II. When intervention is conducted, actual performance is plotted and compared to the trend line; III. Improvement in actual performance over what the trend line predicted can then be reasonably attributed to the intervention.
  • 44. Step 2b: Planning the Intervention (With ROI in Mind) Trend Line Analysis- Advantages and Disadvantages Advantages Disadvantages I. Show quick results I. Uses only one factor to isolate the effect of the intervention (time- before and II. Easy to calculate after implementation) III. Easy to understand II. Thus less accurate and client could be IV. Inexpensive sceptical to accept only an analysis of V. Not difficult to organize (basically just the trend line. monitor changes in metrics) VI. A good first order analysis
  • 45. Step 2b: Planning the Intervention (With ROI in Mind) Example- Not from our central case study… Effect of the intervention COMPLAINTS Average Pre Program 55 Average Sexual Post Program 35 Harassment Prevention Program O N D J F M A M J J A S O N D J F M A M J J A S O TIME
  • 46. Step 2b: Planning the Intervention (With ROI in Mind) Control Groups Technique I. Similar to classical research the control group method, involves an experimental group and a control group; II. The so called experimental group will be the target of our intervention; III. The control group would be a group that is paired with the experimental group, but would not undergo the intervention; IV. Important things to remember with this method are: a. Paring need to be thought about carefully so as not to dilute the impact of the intervention; b. Although not as rigid as classical research key factors that could dilute the impact or assessment of the impact of the intervention need to be controlled.
  • 47. Step 2b: Planning the Intervention (With ROI in Mind) Control Groups- Advantages and Disadvantages Advantages Disadvantages I. More accurate than most techniques I. More complicated II. If the pairing and control of variables are II. Require more planning, negotiation, agreed with the client the outcome has high control and monitoring level of credibility III. As result of above many practitioners are III. With this control groups, given the level of unwilling to use this technique accuracy it is possible to benchmark the ROI IV. Requires absolute agreement with the of certain interventions given a client environment that variables will be predetermined set of variables kept constant for the duration of the ROI evaluation.
  • 48. Step 2b: Planning the Intervention (With ROI in Mind) Schematic Explanation of control groups design 1. Pairing 2. Control Variables Control M1 M2 Group Experimental M1 Intervention M2 Group *M = measurement
  • 49. Step 2b: Planning the Intervention (With ROI in Mind) Looking at pairing and control of variables PAIRING Note: I. Too small then meaningful pairing is not possible – too big then you are not going to be able to control the variables during and after the implementation of your Select your intervention; experimental and control group II. Remember we go for reasonable carefully accuracy therefore you pair the major elements and also control the major variables!! III. The above makes it very important to agree upfront with your client environment what how pairing will be VARIABLES done and which variables will be controlled.
  • 50. Step 2b: Planning the Intervention (With ROI in Mind) So what do you typically pair? Pair for (Primarily) General demography Performance level Leadership stype Nature of work Size Negotiate creafully
  • 51. Step 2b: Planning the Intervention (With ROI in Mind) So what do you typically try to control? Changes in: Changes in: Systems and procedures Recognition and reward General demography Performance level Leadership stype Nature of work Turn over structure Function + Size Negotiate creafully Negotiate creafully
  • 52. Step 2b: Planning the Intervention (With ROI in Mind) Remember your decision about pairing and control of variables…. = (f) (your knowledge and judgement) + agreement with the client environment + practicability
  • 53. Step 2b: Planning the Intervention (With ROI in Mind) Estimation of impact by third parties ( this technique can include an estimation of the perceived impact of the intervention, by participants, participants supervisors, participants managers) This technique provides participants with a questionnaire, and based on the improvement ask them to respond to the following: I. What percent of this improvement can be attributed to the application of skills/techniques/knowledge gained in the training program? II. What is the basis for this estimation? III. What confidence do you have in this estimate, expressed as a percentage? IV. What other factors contributed to this improvement in performance? V. What other individuals or groups could estimate this percentage or determine the amount?
  • 54. Step 2b: Planning the Intervention (With ROI in Mind) Estimation of impact by third parties – how do you increase the confidence level for this technique? I. Individuals who do not respond to the questionnaire or provide non- usable data on the questionnaire are assumed to report no improvement/impact; II. Extreme data and unrealistic claims are omitted from the analysis; III. Only annualized values are used, it is assumed that there are no benefits from the program after the first year of implementation; IV. The confidence level, expressed as a percent, is multiplied by the improvement value to reduce the amount of the improvement by the potential error; V. If value is expressed it is also factored by the confidence value.
  • 55. Step 2b: Planning the Intervention (With ROI in Mind) Estimation Of Impact By Third Parties - Advantages And Disadvantages Advantages Disadvantages I. Easy to implement; I. Less accurate than control groups II. Based on the assumption that participants II. Based on perceptions – could be are capable of determining or estimating regarded as Return on Expectation impact; rather than ROI III. Although an estimate, this value will usually III. Could become victim of “group think” have considerable credibility with management because participants are at the centre of the change or improvement; IV. High face validity.
  • 56. Step 2b: Planning the Intervention (With ROI in Mind) Example from an actual coaching intervention- Not from our central case study… Participants and management agreed which All indicators the respective were pre groups will agreed with the rate client
  • 57. Step 2b: Planning the Intervention (With ROI in Mind) So now we can complete our plan Kirkpatrick Objectives Measures Where Will We In What When Purpose of Level Find Information Format Data For Evaluation Collection and Analysis Impact See previous See HR Budget vis Business Six months ROI and slides previous Actual Report after final Training slides Staff Turn Over course was Evaluation Records HR Report presented Transfer See previous See Audit/Observatio Possibly Period of three Training slides previous n results checklists months Evaluation slides following training Learning See previous See Test scores Tests papers Within one Training slides previous Practical and answer week following Evaluation slides simulation scores sheets training Simulation checklists and grading papers Reaction See previous See Survey results Possibly Within one Training slides previous tabulated and week following Evaluation slides presented in training graphs
  • 58. Step 2b: Planning the Intervention (With ROI in Mind) Note – because the techniques related to isolating the impact of other factors are not used often it is only referred to in this toolkit… a. Trend line analysis of performance data; b. Control groups; c. Participant’s estimation of impact; d. Supervisor’s estimation of impact; With these e. Management’s estimation of impact; techniques we try to isolate the f. Use of previous studies effects of other interventions g. Subordinate’s report of other factors; and then declare the rest as due h. Estimating the impact of other factors; to our effort i. Use of customer input.
  • 59. Step 3 Implementation and Data Gathering (With ROI in Mind)
  • 60. Implementation and Data Gathering (With ROI in Mind) Referring back to our original example (turnover in graduate engineers) – let us assume the following: Correct Correct pairing pairing Correct Correct control of control of variables variables Agreement Agreement Mine A on metrics Mine B on metrics Control Experiential We opted for a control group evaluation method
  • 61. Implementation and Data Gathering (With ROI in Mind) Referring back to our original example (turnover in graduate engineers) – let us assume the following: Impact Transfer -Evaluation completed Six months after the intervention the need for Learning -Evaluation completed replacement of graduates were down by 50% in the experimental Reaction – Evaluation group completed
  • 62. Implementation and Data Gathering (With ROI in Mind) Referring back to our original example (turnover in graduate engineers) – let us assume the following: Impact Transfer -Evaluation completed In the control group the need for replacement of graduate engineers also Learning -Evaluation completed down by 10%...possibly because of anticipation of changes in HR approach Reaction – Evaluation completed
  • 63. Implementation and Data Gathering (With ROI in Mind) Referring back to our original example (turnover in graduate engineers) – let us assume the following: Impact Transfer -Evaluation completed In the control group the need for replacement of graduate engineers also Learning -Evaluation completed down by 10%...possibly because of anticipation of changes in HR approach Reaction – Evaluation completed
  • 64. So because to the method we used we can assign the change in te metric to our intervention, mainly because we controlled major variables. We do this as follows…
  • 65. Step 4 Converting the Change As Result of the Intervention to Value and Calculating ROI
  • 66. Step 4.a Converting the Change As Result of the Intervention to Value Referring back to our original example (turnover in graduate engineers)…this is how it is dome… Identify the unit of In our case study: Value per unit = improvement Number of R 74964 (see slide 12 replacements of AND 13 graduate engineers in the past six months Determine the Original replacements = Original cost of performance level 6 replacement change Six months after the R 449784 intervention = 3 Our latest cost of Improvement of 50% replacement following intervention R 224892 Calculate the 50% saving R 224892 performance improvement
  • 67. Step 4.a Converting the Change As Result of the Intervention to Value Because this is such a vast area to know here is some additional input re calculating the value of change in a metric.. Primary Measurements of Improvement
  • 68. Step 4.a Converting the Change As Result of the Intervention to Value Always try to link to a metric – alternatively use the method of Estimation Of Impact By Third Parties – See slide 55 and 56 (this method is also referred to as Return on Expectation)
  • 69. Step 4.a Converting the Change As Result of the Intervention to Value See whether one can link tangible values to these? OUTPUT TIME Units Produced Equipment Downtime Tons Manufactured Overtime Items Assembled Time to Project Completion Money Collected Processing Time Items Sold Break in Time for New Employees Forms Processed Learning Time Loans Approved Meeting Schedules Inventory Turnover Repair Time Patients Visited Work Stoppages Applications Processed Order Response Tasks Completed Late Reporting Output Per Hour Lost Time Days Productivity QUALITY New Accounts Generated Scrap COSTS Waste Budget Variances Rejects Unit Costs Error Rates Cost By Account Rework Variable Costs Shortages Fixed Costs Product Defects Overhead Cost Deviation From Standard Operating Costs Product Failures Number of Cost Reductions Inventory Adjustments Project Cost Savings Time Card Corrections Accident Costs Percent of Tasks Completed Properly Sales Expense Number of Accidents
  • 70. Step 4.a Converting the Change As Result of the Intervention to Value And these…. WORK HABITS CUSTOMER SERVICE Absenteeism Customer Complaints Tardiness Customer Satisfaction Visits to the Dispensary Customer Dissatisfaction First Aid Treatments Customer Impressions Violations of Safety Rules Customer Loyalty Number of Communication Break-downs Customer Retention Excessive Breaks Customer Value Follow-Up Lost Customers EMPLOYEE DEVELOPMENT/ADVANCEMENT WORK CLIMATE/SATISFACTION Number of Promotions Number of Pay Increases Number of Grievances Number of Learning Programs Attended Number of Discrimination Charges Requests for Transfer Employee Complaints Performance Appraisal Ratings Job Satisfaction Increases in Job Effectiveness Employee Turnover Litigation INITIATIVE/INNOVATION Organization Commitment Implementation of New Ideas Employee Loyalty Successful Completion of Projects Increased Confidence Number of Suggestions Implemented Setting Goals and Objectives New Products and Services Developed New Patents and Copyrights
  • 71. Step 4.a Converting the Change As Result of the Intervention to Value Or determining value on a case by case basis using historical data Legal fee; Actual cost Settlement from records cost Total of Cost 35 complaints R 285,000 Estimated Staff time; additional Management from staff time R 285,000/35 = Cost per complaint Cost of Sexual Harassment Complaint
  • 72. Step 4.a Converting the Change As Result of the Intervention to Value In our case example we did determine the value of the change and can therefore now move on to calculate cost benefit and therefore also ROI
  • 73. Step 4.b Calculating ROI First thing to do now is to determine the cost of the intervention – again back to our main example … Cost element Notes ZAR value (for the sake of our example) Development Costs If you use more than one trainer or one R50,000 trainer on more than one project – pro-rate Program Materials R10,000 Instructor/Facilitator Costs If you use more than one trainer or one R10,000 trainer on more than one project – pro-rate Facilities Costs R10,000 Travel and lodging R10,000 Salaries and time loss of participants Will need to consult HR and line and pro-rate R50,000 Administration costs R5,000 Evaluation costs R 5,000 Total cost of the intervention R150,000
  • 74. Step 4.b Calculating ROI First thing to do now is to determine the cost of the intervention – again back to our main example … Net Program Benefits (R224892- R150,000) Benefit/Cost Ratio = Program Costs (R150,000) Ratio = 0.499928
  • 75. Step 4.b Calculating ROI First thing to do now is to determine the cost of the intervention – again back to our main example … Net Program Benefits (R224892- R150,000) X 100 Benefits ROI = Program Costs (R150,000) 1 Benefit ROI = 50% thus for every R1 I spent on this program you got your money back through saving as well as an additional 50 cents.