Do your stakeholders want to see evidence of the program's impact?
By knowing what works in your learning portfolio, you can repeat successes and eliminate ineffective programs. This method is based on the work of Donald L. Kirkpatrick, Dr J. Phillips, Robert Brinkerhoff and Josh Bersin.
"Credible, data-driven and actionable reporting of your training impact and talent programs are critical for making right investment decisions."
www.greenbookslearning.com
12. No business need for the program Program linked to specific business
needs
No effort to prepare the work
environment to support program
Environment prepared to support
program
ResultsActivity
13. No business need for the program Program linked to specific business
needs
No effort to prepare the work
environment to support program
Environment prepared to support
program
Reporting on program is input
focused
Reporting on program is output
focused
15. Reaction Objectives
1) Find it important to your success on the job
2) Find the case studies relevant and useful
3) Find the learning process fun
16. Learning Objectives
1) Identify the different evaluation levels
2) Align programs to business needs
3) Identify at least 3 ways to collect data
4) Describe at least 2 ways to isolate the effects of the program
5) Calculate ROI given benefits & cost
33. Kirkpatrick Model Phillips ROI Model
Level 1
Level 2
Level 3
Level 4
Accountability
Process Model
Standards
Publications
Usage
Dr. Donald Kirkpatrick Dr. Jack PhillipsLevel 1Developed
Inputs / IndicatorsLevel 0
Level 5
34. • Number of projects
• Number of participants
• Hours
• Request
• Attendance
• Costs (fully loaded)
• Efficiencies
Level 0 - Inputs and Indicators
35. Kirkpatrick Model Phillips ROI Model
ReactionLevel 1
Level 2
Level 3
Level 4
Accountability
Process Model
Standards
Publications
Usage
Dr. Donald Kirkpatrick Dr. Jack PhillipsLevel 1Developed
Inputs / IndicatorsLevel 0
Level 5
38. Kirkpatrick Model Phillips ROI Model
Reaction
Reaction, Satisfaction & Planned
Action
Level 1
Level 2
Level 3
Level 4
Accountability
Process Model
Standards
Publications
Usage
Dr. Donald Kirkpatrick Dr. Jack PhillipsLevel 1Developed
Inputs / IndicatorsLevel 0
Level 5
Learning Learning
39. Kirkpatrick Model Phillips ROI Model
Reaction
Reaction, Satisfaction & Planned
Action
Learning Learning
Behaviour Application
Level 1
Level 2
Level 3
Level 4
Accountability
Process Model
Standards
Publications
Usage
Dr. Donald Kirkpatrick Dr. Jack PhillipsLevel 1Developed
Inputs / IndicatorsLevel 0
Level 5
41. Week 1
Week 2 Week 3
Week 4 Week 5 Week 6 Week 7
Week 8
Pre
Assessment
Classroom & Online Training
ILT (3hours, 5 alternative days)
Online (10 hours, 10 days)
Post Training Support - Email Coach
(2 emails a week & online exercises) Post
Assessment
intermediate
Deutsche Bank training flow
42. Kirkpatrick Model Phillips ROI Model
Reaction
Reaction, Satisfaction & Planned
Action
Learning Learning
Behaviour Application
Results Business Impact
Level 1
Level 2
Level 3
Level 4
Accountability
Process Model
Standards
Publications
Usage
Dr. Donald Kirkpatrick Dr. Jack PhillipsLevel 1Developed
Inputs / IndicatorsLevel 0
Level 5
46. Kirkpatrick Model Phillips ROI Model
Reaction
Reaction, Satisfaction & Planned
Action
Learning Learning
Behaviour Application
Results Business Impact
Isolating the effects of the program
Level 1
Level 2
Level 3
Level 4
Accountability
Process Model
Standards
Publications
Usage
Dr. Donald Kirkpatrick Dr. Jack PhillipsLevel 1Developed
Inputs / IndicatorsLevel 0
Level 5
47. Kirkpatrick Model Phillips ROI Model
Reaction
Reaction, Satisfaction & Planned
Action
Learning Learning
Behaviour Application
Results Business Impact
Isolating the effects of the program
Level 1
Level 2
Level 3
Level 4
Accountability
Process Model
Standards
Publications
Usage
Dr. Donald Kirkpatrick Dr. Jack PhillipsLevel 1Developed
Inputs / IndicatorsLevel 0
ROI (ROE)Level 5
52. Level Current status Evaluation Targets
0 Less then 25% of the programs 100%
1 100% of the workshops conducted. 100%
2 Done for more than 60-70% 90%
3 Less than 10% of the training 50%
4 Not done for any training 30%
5 Not done for any training 15%
64. Kirkpatrick Model Phillips ROI Model
Reaction
Reaction, Satisfaction & Planned
Action
Learning Learning
Behaviour Application
Results Business Impact
Isolating the effects of the program
Level 1
Level 2
Level 3
Level 4
Accountability
Process Model
Standards
Publications
Usage
Dr. Donald Kirkpatrick Dr. Jack PhillipsLevel 1Developed
10 step by step of collecting, analyzing, reporting, using systematic &
methodic
Inputs / IndicatorsLevel 0
Return on InvestmentLevel 5
65. Develop
Objectives of
Solution (s)
Develop
Evaluation
Plans and
Baseline Data
Collect
Data During
Solution
Implementation
Collect
Data After
Solution
Implementation
Level 2: Learning Level 4: Business
Impact
Level 1: Reaction,
Satisfaction, and
Planned Actions
Level 3:
Application/
Implementation
Evaluation Planning Data Collection
66. Isolate the
Effects
Convert Data
to Monetary
Value
Calculate the
Return on
Investment
Tabulate Costs
of Solution
Identify
Intangible
Measures
Generate
Impact
Study
Level 5:
ROI
Intangible Measures
Data Analysis Reporting
67. Kirkpatrick Model Phillips ROI Model
Reaction
Reaction, Satisfaction & Planned
Action
Learning Learning
Behaviour Application
Results Business Impact
Isolating the effects of the program
Level 1
Level 2
Level 3
Level 4
Accountability
Process Model
Standards
Publications
Usage
Dr. Donald Kirkpatrick Dr. Jack PhillipsLevel 1Developed
10 step by step of collecting, analyzing, reporting, using systematic &
methodic
12 guiding principles to be conservative in approach
Inputs / IndicatorsLevel 0
Return on InvestmentLevel 5
69. Kirkpatrick Model Phillips ROI Model
Reaction
Reaction, Satisfaction & Planned
Action
Learning Learning
Behaviour Application
Results Business Impact
Isolating the effects of the program
Level 1
Level 2
Level 3
Level 4
Accountability
Process Model
Standards
Publications
Dr. Donald Kirkpatrick Dr. Jack PhillipsLevel 1Developed
10 step by step of collecting, analyzing, reporting, using systematic &
methodic
12 guiding principles to be conservative in approach
5 books 40 books, 3000 case studies
Inputs / IndicatorsLevel 0
Return on InvestmentLevel 5
87. Aligning needs to business
Learning Needs
Preference Needs
Performance Needs
Business Needs
Payoff Needs Is the problem worth solving?5
4
3
2
1
Needs
88. Pay off need is the financial impact of
the problem
90. Aligning needs to business
Learning Needs
Preference Needs
Performance Needs
What business measures affect the problem? (lag measure)Business Needs
Payoff Needs Is the problem worth solving?5
4
3
2
1
Needs
94. Business Impact Metrics
• Decrease in time to billability
• Decrease in manager/ lead time
spent
• Decrease in time to competency
95. Aligning needs to business
Learning Needs
Preference Needs
What should happen on the job to impact the business measure? (lead)Performance Needs
What business measures affect the problem? (lag measure)Business Needs
Payoff Needs Is the problem worth solving?5
4
3
Needs
97. audience behaviour condition degree
Supervisors should counsel their team members when there is an
unplanned absenteeism at least 9 out of 10 times
98. If you do not have the
confidence of line managers,
no amount of analysis will help
you case.
Source: Mckinsey Oct, 2010
99. Aligning needs to business
Learning Needs
Preference Needs
What skills/ knowledge must be developed to meet the performance need?
What should change in the work environment to impact the business measure? (lead)Performance Needs
What business measures affect the problem? (lag measure)Business Needs
Payoff Needs Is the problem worth solving?5
4
3
2
Needs
100.
101. Aligning needs to business
Learning Needs
Preference Needs How should the solutions be structured for participants satisfaction?
What skills/ knowledge must be developed to meet the performance need?
What should change in the work environment to impact the business measure? (lead)Performance Needs
What business measures affect the problem? (lag measure)Business Needs
Payoff Needs Is the problem worth solving?5
4
3
2
1
Needs
103. Aligning needs to business
Learning Needs
Preference Needs How should the solutions be structured for participants satisfaction?
What skills/ knowledge must be developed to meet the work performance need?
What should change in the work environment to impact the business measure? (lead)Performance Needs
What business measures affect the problem? (lag measure)Business Needs
Payoff Needs Is the problem worth solving?5
4
3
2
1
Needs
104. Aligning objectives to needs
Learning Objectives
Reaction Objectives How should the participants react?
What will the participants learn at the end of the program?
How should the audience behave under given condition to an acceptable degree (abcd)?Application Objectives
How much of the increase/ decrease should take place in your business measure?Impact Objectives
ROI Objectives What is the expected ROI?
Program
105. Aligning evaluation to objectives
Learning
Reaction How do I collect the participants reactions to the program?
How will I test if the participants have acquired the skills / knowledge?
How will I track if my participants are applying the learning on the job? (lead measure)Application
Where do I get the increase/ decrease/ cost avoidance data? (lag measure)Impact
ROI What is the program ROI?5
4
3
2
1
Evaluation
107. Learning Needs
Preference Needs
Performance Needs
Business Needs
Payoff Needs
5
4
3
2
1
Absenteeism is causing 15,00,000
monthly
Unplanned absenteeism is 9%
Discussion between team
members and supervisor are not
occurring
Counseling and discussion skills
Counseling skills must be practical and should
happen through ILT & Coaching
112. ROI Objectives
Learning Needs
Preference Needs
Performance Needs
Business Needs
Payoff Needs
5
4
3
2
1
Discussion between team
members and supervisor are not
occurring
Counseling and discussion skills
Counseling skills must be practical and
should happen immediately
Reaction Objectives
Learning Objectives
Application Objectives
Impact Objectives
Absenteeism is causing 15,00,000
monthly
Unplanned absenteeism is 9%
113. Aligning objectives to needs
Learning Objectives
Reaction Objectives How should the participants react?
Application Objectives
Impact Objectives
ROI Objectives
Program
114. ROI Objectives
Learning Needs
Preference Needs
Performance Needs
Business Needs
Payoff Needs
5
4
3
2
1
Discussion between team
members and supervisor are not
occurring
Counseling and discussion skills
Counseling skills must be practical and
should happen immediately
4/5 on the practicality of the
program and relevance to job
Reaction Objectives
Learning Objectives
Application Objectives
Impact Objectives
Absenteeism is causing 15,00,000
monthly
Unplanned absenteeism is 9%
115. Aligning objectives to needs
Learning Objectives
Reaction Objectives
What will the participants learn at the end of the program?
Application Objectives
Impact Objectives
ROI Objectives
Program
116. ROI Objectives
Learning Needs
Preference Needs
Performance Needs
Business Needs
Payoff Needs
5
4
3
2
1
Discussion between team
members and supervisor are not
occurring
Counseling and discussion skills
Counseling skills must be practical and
should happen immediately
4/5 on the practicality of the
program and relevance to job
Participants will learn 3 key
counseling skills
Reaction Objectives
Learning Objectives
Application Objectives
Impact Objectives
Absenteeism is causing 15,00,000
monthly
Unplanned absenteeism is 9%
117. Aligning objectives to needs
Learning Objectives
Reaction Objectives
How should the audience behave under given condition to an acceptable degree (abcd)?Application Objectives
Impact Objectives
ROI Objectives
Program
118. ROI Objectives
Learning Needs
Preference Needs
Performance Needs
Business Needs
Payoff Needs
5
4
3
2
1
Discussion between team
members and supervisor are not
occurring
Counseling and discussion skills
Counseling skills must be
practical and should happen
immediately
4/5 on the practicality of the
program and relevance to job
Participants will learn 3 key
counseling skills
Counseling discussion conducted in
95% situations when an unexpected
absence occurs
Reaction Objectives
Learning Objectives
Application Objectives
Impact Objectives
Absenteeism is causing 15,00,000
monthly
Unplanned absenteeism is 9%
119. Aligning objectives to needs
Learning Objectives
Reaction Objectives
Application Objectives
How much of the increase/ decrease should take place in your business measure?Impact Objectives
ROI Objectives
Program
120. ROI Objectives
Learning Needs
Preference Needs
Performance Needs
Business Needs
Payoff Needs
5
4
3
2
1
Discussion between team
members and supervisor are not
occurring
Counseling and discussion skills
Counseling skills must be
practical and should happen
immediately
4/5 on the practicality of the
program and relevance to job
Participants will learn 3 key
counseling skills
Counseling discussion conducted in
95% situations when an unexpected
absence occurs
Reduce absenteeism to 5%
Reaction Objectives
Learning Objectives
Application Objectives
Impact Objectives
Absenteeism is causing 15,00,000
monthly
Unplanned absenteeism is 9%
121. Aligning objectives to needs
Learning Objectives
Reaction Objectives
Application Objectives
Impact Objectives
ROI Objectives What is the expected ROI?
Program
122. ROI Objectives
Learning Needs
Preference Needs
Performance Needs
Business Needs
Payoff Needs
5
4
3
2
1
Discussion between team
members and supervisor are not
occurring
Counseling and discussion skills
Counseling skills must be
practical and should happen
immediately
4/5 on the practicality of the
program and relevance to job
Participants will learn 3 key
counseling skills
Counseling discussion conducted in
95% situations when an unexpected
absence occurs
Reduce absenteeism to 5%
ROI of 20%
Reaction Objectives
Learning Objectives
Application Objectives
Impact Objectives
Absenteeism is causing 15,00,000
monthly
Unplanned absenteeism is 9%
130. Aligning evaluation to objectives
Learning
Reaction How do I collect the participants reactions to the program?
How will I test if the participants have acquired the skills / knowledge?
How will I track if my participants are applying the learning on the job? (lead measure)Application
Where do I get the increase/ decrease/ cost avoidance data? (lag measure)Impact
ROI What is the program ROI?5
4
3
2
1
Evaluation
131. Sales Skills - Tata Sky
Learning Needs
Preference Needs Reaction
Learning
Application
Impact
ROI
Reaction Objectives
Learning Objectives
Application ObjectivesPerformance Needs
Impact ObjectivesBusiness Needs
Payoff Needs ROI Objectives
End HereStart Here
5
4
3
2
5
4
3
2
1 1
Increase Sales
Increase EPRS
Increase walkin conversion
Profiling of the customer,
Standardization in customer
demo, & transaction
Greeting
Asking right questions to the prospects
to understand the wants and needs,
Match the needs with features &
benefits
ILT and role plays 4/5 on the practicality of the
program and relevance to
job
Participants will learn key
interaction & influencing skills
Use the skills consistently, which is 100% of the
times
+ve ROI
Track business dashboards
every month
Calculate ROI, BCR
Increase walk-in conversion ratio
Increase the subscription recharge by 10%
Increase Sales by 10%
Action plan activity sheet
Facilitator assessment through role
plays
Reaction and planned action through a
questionnaire at the end of the program
132. Learning Needs
Preference Needs
Performance Needs
Business Needs
Payoff Needs
5
4
3
2
1
Increase Sales
Increase EPRS
Profiling of the customer,
Standardization in customer
demo, & transaction
Greeting
understand the wants and needs
Influencing
ILT and role plays
4/5 on the practicality and
relevance
Participants will learn key
interaction & influencing
skills
Use 100% of the skills
consistently with every customer
interaction
+ve ROI
Increase Sales by
Increase the subscription recharge by
133. V Model – Tata Sky
Learning Needs
Preference Needs Reaction
Learning
Application
Impact
ROI
Reaction Objectives
Learning Objectives
Application ObjectivesPerformance Needs
Impact ObjectivesBusiness Needs
Payoff Needs ROI Objectives
End HereStart Here
5
4
3
2
5
4
3
2
1 1
Increase Sales
Increase EPRS
Profiling of the customer,
Standardization in customer
demo, & transaction
Greeting
Asking right questions to the prospects
to understand the wants and needs,
Match the needs with features &
benefits
ILT and role plays 4/5 on the practicality of the
program and relevance to
job
Participants will learn key
interaction & influencing skills
Use the skills consistently, which is 100% of the
times
+ve ROI
Track business dashboards
every month
Calculate ROI, BCR
Increase the subscription recharge by 10%
Increase Sales by 10%
Action plan activity sheet
Facilitator assessment through role
plays
Reaction and planned action through a
questionnaire at the end of the program
134. Each level includes:
• Data Sources
• Data Collection
• Key Questions
• Key Issues
Collect Data After
Solution
Implementation
Isolate
the Effects
of the Solution
Convert Data To
Monetary
Value
Calculate
The Return On
Investment
Tabulate
Costs of
Solution
Identify
Intangibles
Significant Influences
• Policy Statement
• Procedures and Guidelines
• Staff Skills
• Management Support
• Technical Support
• Organizational Culture
Phillips ROI Methodology TM
Develop
Report and
Communicate
Results
Level 4 Level 3 Level 3 and 2 Level 1
Level 3
Level 4
Level 5
Level 5
Results-Based Approach
Intangible
Benefits
Specify Skill/
Knowledge
Deficiencies of
Affected
Population
Training
Required
?
Level 2
Problem/
Opportunity
Present or
Anticipated
Identify Job
Performance
Needs, Gaps,
and Why
Identify
Solutions
Identify
Preferences
Develop
Objectives/
Evaluation
Strategy
Identify Transfer
Strategy Options
and L-2 & L-3
Support For All
Stakeholders
Identify
Business,
Needs,
Gaps, and
Stakeholder
s
Design
Solution and
Stakeholder
Components
Consider
Resources/
Logistics
Delivery
Develop
Content/
Materials
Conduct/
Implement
Solution
Level 1
Level 2
Implement
Pre-activity
Implement
Transfer
Strategy
- Solution
- Pre-work Activity/Transfer Strategy
Page 3.9
139. Data Items
(Usually
Level 4
Method for
Isolation
Methods of
Converting
Data to
Monetary
Values
Cost Categories Intangible
Benefits
Communication
Targets for Final
Report
Other
Influences/
Issues
During
Application
Monthly
conversion
ratio
Participant
estimate
(ROMs &
ROI)
Standard
Value.
(1)Facilitation Fees
(2)Program Materials
(3)Market survey Fees
(4)Facility costs
(5)Participants
salaries
ROMs
ROIs
Exec. V P
Retail
operations
Any
change in
market
factor
ROI Analysis Plan– Tata Sky
140. Develop
Objectives of
Solution (s)
Collect
Data During
Solution
Implementation
Collect
Data After
Solution
Implementation
Evaluation Planning Data Collection
Level 2: Learning
Level 1: Reaction,
Satisfaction, and
Planned Actions
142. 1. Relevance to work
2. Important to success
3. Would recommend to others
4. Amount of new information
5. Method of delivery
6. Learning Environment
7. Planned Action
Level 1 Target Areas
143.
144. Number of Ratings
Rating Course 1 Course 2
5 (excellent)
4 (good)
3 (fair)
2 (poor)
1 (dismal)
Average Satisfaction 3.7 3.9
148. 1. Test
2. Simulation
3. Business games
4. In-basket
5. Case Studies
6. Role playing
7. Assessment center
8. Self, team, facilitator assessment
Techniques for Measuring
Learning
149. Develop
Objectives of
Solution (s)
Develop
Evaluation
Plans and
Baseline Data
Collect
Data During
Solution
Implementation
Collect
Data After
Solution
Implementation
Level 4:
Business Impact
Level 3:
Application/
Implementation
Evaluation Planning Data Collection
150. 1. Skills - Demonstration of the
new skills on the job
2. Knowledge
3. Capacity – Demonstrate ability
on the job
4. Awareness – Demonstrate the
required behaviors on the job
Level 3 Target Areas
153. • Manager does not support the learning
• Process does not support learning
• No opportunity to use the skills
• No time to implement the skills
• Didn’t learn anything that could be applied to work
• Resources are not available to implement the skills
• Dint see a need to implement
• No body cares for the change
154. Increase / Decrease / Cost
avoidance in:
1. Hard data
2. Soft Data
Level 4 Target Areas
155. L4 - Time to Billability
May 2013 : 136 days
May 2014: 57 days
No. of days saved per CM is 78.8 days
156. L4 - Time to Billability
Total no. of days saved by May 2014 Campus Mind (78) to
become billable is 6146 days
157. A. Survey
B. Test
C. Questionnaire
D. Interview
E. Focus Group
F. Observation
G. Performance Records
Data Collection Instruments
159. 20% analysis
80% data capture
80%
analysis
20% data capture
Measurement as a project Measurement as a process
160. Isolate the
Effects
Convert Data
to Monetary
Value
Calculate the
Return on
Investment
Tabulate Costs
of Solution
Identify
Intangible
Measures
Generate
Impact
Study
Level 5:
ROI
Intangible Measures
Data Analysis Reporting
Step
5
162. 1. Trend line analysis of performance data
2. Use of a control group arrangement
3. Estimate of the program’s impact (percent)
4. Use of experts/previous studies
164. 20%
10%
J F M A M J J A S O N D J
REJECT
RATES
18.5% Pre Program Six-Month Average
Projected Average —
Using Pre Data as a
Base 14.5%
7% Post Program
Six-Month
Average
CPI Program Conducted
MONTHS
165. Established trend should continue on the same path
No other variables and influences should entered the
process after the training
174. 1. Describe the task and explain how it will be used
2. Identify and list all potential factors
3. Estimate the contribution for each factors
4. Obtain the confidence level for error correction
5. Calculate the attributable data
175. Dear Colleague
MindTree and GreenBooks, an expert training evaluation company,
have conducted a study to measure the results of the training that the
Minds of May 2014 underwent in Orchard Program.
The study showed the following result:
Number of days saved per mind to become billable – 63 days
(average comparison data of May 2013 and 2014)
Upon interacting with the heads of Recruitment, Delivery and CDM,
we have identified that the following factors could have potentially
contributed to the save in number of days for the Mind's billability:
176. Project Demand
• Delivery head indicated that the project demand of 2014 was higher than 2013
Quality of recruitment process
• College candidates of relevant disciplines (Computer Science, IT) only were
considered for recruitment. Unlike the previous batch, students from other
disciplines (Civil, Automation, Mechanical, Biotech etc) were not considered in this
batch
• Panel Briefing emphasized quality of candidates over quantity
• Selection criteria was stringent and adhered to
Orchard Learning
• 3 months of training led by MindTree's Project Leads
• Course was designed with project and case application embedded
• Course content was created based on inputs from project managers
• Assessments process was rigorous and included case application. Only those Minds
who cleared were allowed into projects
Project assimilation and on the job learning
• Knowledge assimilated as a result of working on the project and learning from
peers, leads etc.
Others
• Leads were assigned KPI to absorb more Minds (c1) into their team
• There was a focus drive to increase billability of the Minds (c1)
• Management attention to reduce billability time
177. Business Impact: 6146 days
Project Demand: 20.25%
Quality of recruitment process: 19.25%
Orchard Learning: 18.25%
Project assimilation and on the job learning: 18.71%
Others: 4%
178. Sales %
Attributable to
training
Confidence Level
ERPS %
Attributable to
training
Confidence
Level
Total Average for
the Group
42% 86% 42% 88%
Corrected Group average for sales 36% of the increase in Sales of Boxes
Corrected Group average for
Subscription Recharge(EPRS)
37% of Increase in Monthly Recharge( EPRS)
Tata Sky used Supervisors Estimates
181. Isolate the
Effects
Convert Data
to Monetary
Value
Calculate the
Return on
Investment
Tabulate Costs
of Solution
Identify
Intangible
Measures
Generate
Impact
Study
Level 5:
ROI
Intangible Measures
Data Analysis Reporting
Step 6
182. • Converting output to contribution - standard value (profit/savings)
• Converting the cost of quality - standard value
• Converting employee’s time (using compensation)
• Using historical costs/savings
• Using internal and external experts
• Using data from external databases/studies
Methods of converting data to monetary
value
184. Exit cost of previous employee Lost productivity
Recruiting cost Quality problems
Employee cost Customer dissatisfaction
Orientation cost Loss of expertise/know
Training cost Supervisor’s time for turnover
Wages and salaries while training Temporary replacemen
Parameters of a Turnover
185. Converting Data to Monetary Values:
Step 1 Define the unit of measure
(Increase in sales, decrease in absenteeism, time saving)
Step 2 Determine the value of each unit
One unit of sales, one cost of an absentee, one hour of time saving)
Step 3 Calculate the change in performance
Change during the reporting period
Step 4 Determine the annual amount of change
Step 5 Calculate the total value of improvement
The value of one unit of improvement multiplied by the amount of annual change
during the reporting period
186. Converting Data to Monetary Values:
Evaluation of a Team-Building Program in a Manufacturing Plant
Step 1 Define the unit of measure
Unit of measure = one less grievance
Step 2 Determine the value of each unit
Using internal experts on the labour relations staff, the cost of an average
grievance was estimated to be 18,000, considering time and direct cost
Step 3 Calculate the change in performance
Six months after the program, total grievances per month had declined by 10.
Supervisors isolated the effects of the program, determining that 7 of the 10
grievance reductions were because of the program
Step 4 Determine the annual amount of change
Using the adjusted value of 7 per month yields an annual improvement of 84
Step 5 Calculate the total value of improvement
Annual value = 84 x 18,000
= 15, 12,000
187. Converting Data to Monetary Values:
Winning Every Walk-in – Tata Sky
Step 1 Define the unit of measure
One new customer,
Subscription recharge for a month
Step 2 Determine the value of each unit
Average revenue earned is Rs. 300 for a set top box(Finance dept. worked out).
Increased subscription recharge per month= Rs. 21,27,878 (after deduction 70%
cost)
Step 3 Calculate the change in performance
Average increment in sales boxes is 1016. Out of which 36% was attributed to
training.
(1016 x 36)/ 100 = 365
The percentage increase in subscription recharge attributable to training is 37%.
(2127878 x 37)/ 100 = 7,87,315
Step 4 Determine the annual amount of change
Annual value of increased sales in the set top box : 300 x 365 x 12 = 13,14,000
Annual value of increased in subscription recharge : 7,87,315 x 12 = 94,47,780
Step 5 Calculate the total value of improvement
188. Converting Data to Monetary Values:
Mindtree
Step 1 Define the unit of measure
Time to billability reduced in days
Step 2 Determine the value of each unit
Each day of billability is Rs. 9997
Step 3 Calculate the change in performance
No. off days days saved is 6146. Out of which 18% was attributed to training.
1107 days were attributed to training
Step 4 Determine the (annual) amount of change
1107 days in a year
Step 5 Calculate the total value of improvement
Total benefit : 1107 * 9997 = 1,10,66,679
191. To Convert or Not to Convert
No
Is there
a
standard
value?
Yes
Is there a
Method to
get there?
Move to
Intangible
Benefits
Can we
get there
with
minimum
resources
?
No
Yes
No No
Can we
convince
executive of
credibility in
2 minutes?
Convert
data and
add to
numerator
Yes Yes
Add to
numerator
Move to
Intangible
Benefits
Move to
Intangible
Benefits
192. Isolate the
Effects
Convert Data
to Monetary
Value
Calculate the
Return on
Investment
Tabulate Costs
of Solution
Identify
Intangible
Measures
Generate
Impact
Study
Level 5:
ROI
Intangible Measures
Data Analysis Reporting
Step
7
193. Fully load the cost
• Needs Analysis cost
• Development cost
• Delivery cost
• Evaluation cost
• Others
194. S. No The Cost category
Value( In
Indian
Rupees)
1. Facilitation fees for the Master trainer 2,00,000
2. The cost for the content 12,00,000
3. The cost for the Market survey 25,000
4.
Participants Salaries(As per managerial grades):
M1-2 Participants( @ Rs.10000 per day) 40,000
M2 – 2 Participants( @ 7000 per day) 28,000
M3 – 2 Participants( @ 4000 per day) 16,000
M4 – 10 Participants( @ 2800 per day) 56,000
M5 – 22 Participants ( @ 1400 per day) 61,600
5.
Cost of internal venue for the six batches done for the Company staff
(ROIs and the ROMs). This included rent of venue and food.
1,50,000
6.
Cost of the 25 training batches for the Experience Zone Employees (@
Rs.25000 per batch).This included rent of venue and food.
6,25,000
7.
Cost of 6 refresher batches – required to tackle the employee churn (@
Rs.25000 per batch).
1,50,000
Total Costs for the training program 25,51,600
Fully loaded Cost – Tata Sky
195. Isolate the
Effects
Convert Data
to Monetary
Value
Calculate the
Return on
Investment
Identify
Intangible
Measures
Generate
Impact
Study
Level 5:
ROI
Intangible Measures
Data Analysis Reporting
Step
8
196. Benefits/ Cost Ratio =
Program Benefits
Program Cost
ROI =
Program Benefits – Program
Costs
Program Costs
X 100
Calculate BCR & ROI
197. Program Benefits = 1,07,58,691
Program Costs = 25,51,600
Benefits/ Cost Ratio =
1,07,58,691
2551600
= 4.21
ROI =
1,07,58,691 – 25,51,600
25,51,600
= 321%X 100
198. Program Benefits = 1,10,66,679
Program Costs = 78,00,000
Benefits/ Cost Ratio =
1,10,66,679
78,00,000
= 1.41
ROI =
1,10,66,679 – 78,00,000
78,00,000
= 41%X 100
199. Isolate the
Effects
Convert Data
to Monetary
Value
Calculate the
Return on
Investment
Tabulate Costs
of Solution
Identify
Intangible
Measures
Generate
Impact
Study
Level 5:
ROI
Data Analysis Reporting
Step
201. • The confidence of the Experience zone employee increased due to the training. This
reflected in all the activities of the store. The morale of the experience zone employee as
well that of the owner increased. The effect was noticeable but since we could not
measure the difference in confidence pre and post training, this was reported as an
intangible benefit.
• The better customer management at the stores reduced the incoming call load at the
contact center. As we could not use any method to isolate the effect we decided to keep it
as an intangible method.
• Better handling of customers during the time of sale also ensured that they recharged for
the first month more regularly. This reduced the casual churn of the customers due to not
recharging their accounts in time. Here again since we could not find a way to isolate this
particular effect of training we decided to list this as an intangible benefit.
• The ROIs got skilled on the job coaching. This will help in better channel management.
Intangibles – Tata Sky
202. Isolate the
Effects
Convert Data
to Monetary
Value
Calculate the
Return on
Investment
Tabulate Costs
of Solution
Identify
Intangible
Measures
Generate
Impact
Study
Level 5:
ROI
Intangible Measures
Data Analysis Reporting
Step 10
203. • General information
• Methodology for impact study
• Data analysis
• Costs
• Results
• Barriers and enablers
• Summary of findings
• Conclusions and recommendations
• Exhibits
Complete Report