Training expense represents a substantial investment in training resource. This dissertation details research on the business impact of a leadership-training program using the return on investment (ROI) methodology. The primary objective of this study was the determination if there were positive financial impacts of a leadership-training program on a business and to present a verifiable and valid, substantial ROI with meaningfulness.
Doctoral Disseration: "Examining the Impact on Business Results Through Post-Training ROI"
1. Examining the Impact of Training on Business Results
Through Post-Training ROI
Dissertation
Submitted to Northcentral University
Graduate Faculty of the Department of Business and Management
in Partial Fulfillment of the
Requirements for the Degree of
DOCTOR OF PHILOSOPHY
by
JACK L. KULES
Prescott Valley, Arizona
May 2008
2. APPROVAL
Examining the Impact of Training on Business Results
Through Post-Training ROI
by
Jack L. Kules
Approved by:
__________________________________________ ________________
Chair: Thomas Driver, Ph.D.
__________________________________________ ________________
Member: David Moody, Ph.D.
__________________________________________ ________________
Member: William Shriner, Ph.D.
Certified by:
__________________________________________ ________________
School Chair: Freda Turner, Ph.D.
3. ABSTRACT
Examining the Impact of Training on Business Results
Through Post-Training ROI
by
Jack L. Kules
Northcentral University, May 2008
Training expense represents a substantial investment in training resource.
This dissertation details research on the business impact of a leadership-training
program using the return on investment (ROI) methodology. The primary
objective of this study was the determination if there were positive financial
impacts of a leadership-training program on a business and to present a
verifiable and valid, substantial ROI with meaningfulness. Training application
and effectiveness were measured through four research tests, and ROI results
and relevance were measured using two additional research tests. Questionnaire
responses and action plan information was examined from 48 employees (from a
target population of about 65) who went through a 15-hour strategic leadership
training program. The training was found to be both effective and successful in
teaching and developing strategic leadership concepts. Perhaps the most
important and tangible indication of successes came directly from the company.
Based on the study’s results, there was no meaningful difference of learning by
the four organizational groups and that all participants have a net positive impact
on business.
iii
4. ACKNOWLEDGEMENTS
When you begin a learning journey such as the one represented by this
dissertation, as the researcher, you think you know where you are headed but
you cannot be sure of the final destination. The experience of completing this
dissertation has been challenging and rewarding. Not only did it foster a sense
of accomplishment and contribution to my field of study but it also allowed me
to meet and become close to some brilliant people.
I thank the participants in this research who gave their generous input
and support to this project. Likewise, I thank ―SP‖ for allowing me to use many
of their resources to make this dream possible.
A special thank you is extended to my dissertation committee. Dr.
Thomas Driver (Committee Chair), Dr. David Moody, and Dr. William Shriner
supported me all the way through the dissertation process and their direction,
suggestions, and concerns during this project made this journey an exceptional
one.
Finally, I want to thank my wife, Bridget, for all the love, patience,
understanding, and support that she gave me over the past four years—without
which I would never have realized this life-long dream becoming a reality. I am
truly and deeply indebted to her.
iv
5. TABLE OF CONTENTS
APPROVAL ........................................................................................................ ii
ABSTRACT ....................................................................................................... iii
ACKNOWLEDGEMENTS ................................................................................. iiv
TABLE OF CONTENTS ..................................................................................... v
LIST OF TABLES ............................................................................................. vii
LIST OF FIGURES ............................................................................................ ix
CHAPTER I: INTRODUCTION .......................................................................... 1
Statement of the Problem............................................................................. 2
Definition of Key Terms ................................................................................ 4
Brief Review of Related Literature ................................................................ 5
Highlights and Limitations of Methodology ................................................... 6
Limitations of the Study ................................................................................ 7
Research Expectations ................................................................................ 8
CHAPTER II: REVIEW OF RELATED LITERATURE ...................................... 10
How Much Is Performance Improvement Really Worth? ............................ 10
Using ROI Forecasting to Develop a High-Impact, High-Value Training
Curriculum ............................................................................................ 12
Measuring Return on Investment for a Mandatory Training Program ......... 14
Resisting Measurement: Evaluating Soft Skills Training for Senior Police
Officers ................................................................................................. 16
A Preprogram ROI for Machine Operator Training ..................................... 17
Getting Results With Interpersonal Skills Training ..................................... 19
Training’s Contribution to a Major Change Initiative ................................... 21
ROI Case Studies ...................................................................................... 23
CHAPTER III: METHODOLOGY ..................................................................... 26
Overview .................................................................................................... 26
Restatement of the Problem....................................................................... 27
Statement of Hypotheses ........................................................................... 27
Description of Research Design ................................................................. 29
Operational Definition of Constructs and Key Variables ............................. 34
v
6. Description of Materials and Instruments ................................................... 35
Selection of Subjects .................................................................................. 37
Procedures ................................................................................................. 38
Discussion of Data Processing................................................................... 44
Methodological Assumptions and Limitations............................................. 46
Ethical Assurances ..................................................................................... 47
CHAPTER IV: FINDINGS ................................................................................ 50
Overview .................................................................................................... 50
Findings...................................................................................................... 51
Analysis and Evaluation of Findings ........................................................... 71
Summary .................................................................................................... 76
CHAPTER V: SUMMARY, CONCLUSIONS AND RECOMMENDATIONS ..... 77
Summary .................................................................................................... 77
Conclusions................................................................................................ 84
Recommendations ..................................................................................... 93
REFERENCES ................................................................................................ 96
APPENDICES ................................................................................................101
Appendix A ............................................................................................... 102
Appendix B ............................................................................................... 108
Appendix C............................................................................................... 110
Appendix D............................................................................................... 118
Appendix E ............................................................................................... 122
Appendix F ............................................................................................... 124
Appendix G .............................................................................................. 131
Appendix H............................................................................................... 136
Appendix I ................................................................................................. 140
vi
7. LIST OF TABLES
Table 1: ROI Case Studies .............................................................................. 23
Table 2: Demographics of the Population ........................................................ 50
Table 3: Mean and Standard Deviation for Questionnaire Responses
Average Across Four Classes .................................................................... 51
Table 4: Mean and Standard Deviation for the 15 Objectives in Question 1
Across Four Classes .................................................................................. 52
Table 5: Mean and Standard Deviation for the Six Elements in Question 2
Across Four Classes .................................................................................. 55
Table 6: Mean and Standard Deviation for the Five Skill Areas in Question
3 Across Four Classes ............................................................................... 57
Table 7: Mean and Standard Deviation for the 14 Topics in Question 4
Across Four Classes .................................................................................. 58
Table 8: Metrics for Action Plans ..................................................................... 63
Table 9: Action Plan Topic Selection ............................................................... 63
Table 10: Action Plan Input from Selection ...................................................... 65
Table 11: Individual Costs for Strategic Leadership Program .......................... 68
Table 12: Total Program Costs, By Class ........................................................ 69
Table 13: One-Way ANOVA of Perceived Relevance of the 15 Objectives
by Organizational Group ............................................................................ 71
Table 14: One-Way ANOVA of Perceived Relevance of the Six Elements
by Organizational Group ............................................................................ 72
Table 15: One-Way ANOVA of Perceived Relevance of the Five Skill
Areas by Organizational Group .................................................................. 73
Table 16: One-Way ANOVA of Perceived Relevance of the 13 Topics by
Organizational Group ................................................................................. 74
Table 17: Mean Time to Action Plan Completion Based on Class................... 74
vii
8. Table 18: One-Way ANOVA of Mean Time to Action Plan Completion
Across the Four Classes ............................................................................ 75
Table F1: Question Data Table From Questionnaires, Question 1 ................ 125
Table F2: Question Data Table From Questionnaires, Question 2 ................ 127
Table F3: Question Data Table From Questionnaires, Question 3 ................ 128
Table F4: Question Data Table From Questionnaires, Question 4 ................ 129
Table G1: ROI Data Table from Action Plans ................................................ 132
Table H1: Values of One-Way ANOVA of Question 1 ................................... 137
Table H2: Values of One-Way ANOVA of Question 2 ................................... 138
Table H3: Values of One-Way ANOVA of Question 3 ................................... 138
Table H4: Values of One-Way ANOVA of Question 4 ................................... 139
Table I1: Mean ROI Accros Four Classes ..................................................... 141
Table I2: Values of One-Way ANOVA of ROI Results ................................... 142
viii
9. LIST OF FIGURES
Figure 1. Question 1 Breakdown of Questionnaire Responses for Class 1 ..... 54
Figure 2. Question 1 Breakdown of Questionnaire Responses for Class 2 ..... 54
Figure 3. Question 1 Breakdown of Questionnaire Responses for Class 3 ..... 55
Figure 4. Question 1 Breakdown of Questionnaire Responses for Class 4 ..... 55
Figure 5. Question 2 Breakdown of Questionnaire Responses for Classes 1
and 2 .......................................................................................................... 57
Figure 6. Question 2 Breakdown of Questionnaire Responses for Classes 3
and 4 .......................................................................................................... 57
Figure 7. Question 3 Breakdown of Questionnaire Responses for Classes 1
and 2 .......................................................................................................... 59
Figure 8. Question 3 Breakdown of Questionnaire Responses for Classes 3
and 4 .......................................................................................................... 59
Figure 9. Question 4 Breakdown of Questionnaire Responses for Class 1 ..... 61
Figure 10. Question 4 Breakdown of Questionnaire Responses for Class 2 ... 61
Figure 11. Question 4 Breakdown of Questionnaire Responses for Class 3 ... 62
Figure 12. Question 4 Breakdown of Questionnaire Responses for Class 4 ... 62
ix
10. 1
Chapter I: Introduction
Training expenses make up a substantial portion of the budget of an
organization, and have come to be seen as an investment in training resources
(Phillips, 2001). Large training expenditures and the need to show value are two
of the primary drivers that have set in motion an increased emphasis on return on
investment (ROI). Attention to ROI is rapidly becoming a central concern of
organizations. Executives are showing an increased interest in ROI, and have
become mindful of how training budgets have grown with limited or no
accountability (Bartram, 1999; Rothwell, 2003). Executives are now demanding a
return on investment for these programs. Further illustrating its prominence,
numerous case studies (Phillips, 1994; Phillips, 1997; Phillips, 2001) have used
return on investment to validate training’s contribution to business results.
Training budgets can be very large and now have the full attention of
executives. The costs can be immense. IBM has a training budget of about $1
billion, and Kinko’s training budget is over $30 million—or 6% of Kinko’s total
payroll (Phillips, 2001). Regardless of the measurement methodology—total
budget, expenditure per employee, percentage of payroll, percentage of
revenue—a large training budget elicits additional evaluation and measurement.
Executives are now demanding increased accountability for the increasing
training expenditures. The use of the ROI methodology offers a view of training
that reflects the bottom line (Phillips, 2002).
The debate as to what should be measured and which results provide the
best evidence of training success will continue, and no measurement has been
11. 2
clearly proven the most reliable. A valid system would employ a balanced set of
measures that take into consideration trainee preferences, learning retention,
learning application, changes in business measures, and actual ROI (Phillips,
1997). This need for balanced measures is the major driver of ROI methodology,
as it measures financial impact along with other important concerns.
Statement of the Problem
The problem addressed in this study was the determination if there were
positive financial impacts of a leadership-training program on a business and to
present a verifiable and valid ROI with meaningfulness. SP had a major need to
determine the value and applicability of its leadership training to the company for
current and future leaders. Therefore, a study of this nature was proposed and
was based on the hypotheses and research questions addressing the differences
in various learning relationships across four organizational groups in a targeted
company. There were many studies (Phillips, 1994; Phillips, 1997; Phillips, 2001)
that indicated a positive ROI, but the lack of a statistically sound approach in
noted studies (see Review of Related Literature for examples) hinders verifying
the reliability of the data in the studies.
Using the ROI methodology to examine the value of training, when
developed, analyzed and reported with meaningful data, will support an
organization’s business success (the ROI methodology will be covered in
Chapter 3). It can verify or validate that the training initiatives are meeting the
needs of the business and having a positive impact on the bottom line of the
business. This methodology has already been used in numerous case studies
12. 3
(Phillips, 1994; Phillips, 1997; Phillips, 2001) to validate the contribution of
training to business results.
Specialty Pharmaceutical (SP)—the fictitious name providing anonymity to
the organization where this study took place—was the organizational body used
for this research. SP opened three new facilities with due to its substantial growth
and trained a group of key current and potential supervisors and managers in
leadership skills. SP needed to change its predominant method of autocratic,
dictatorial style of management to a leadership style that lends itself to a higher
performance work force. Most of the supervisory staff had modeled their
leadership after the former style, as they had been exposed to it as a worker.
The leadership-training program was designed to promote the essential
skills of creativity development, motivation, delegation, communication, and
decision-making. Emphasis was placed on balancing the human-relations side of
management with the drive for results.
The time-spaced format allowed for the real-world application of concepts
between sessions as well as a forum to report on the results achieved. Each
session contained practical application projects that corresponded to the
participants’ responsibilities—so that the projects completed during the program
would result in improved performance
During the program, each participant completed a one-year management
plan (action plan). This project required follow-up actions to initiate actual cost
savings and improvements in the work setting. This positive form of
accountability was to ensure long-lasting and measurable results.
13. 4
Five research questions were formulated to address the problem which
drove the hypotheses and resulting testing:
1. What differences, if any, existed in the perceived relevance of the 15
objectives across the four classes at SP?
2. What differences, if any, in the perceived relevance of the six elements
of the job existed across the four classes at SP?
3. What differences, if any, in the perceived relevance of the five skill
areas existed across the four classes at SP?
4. What differences, if any, in the perceived relevance of the 13 topics in
one’s own work or that of the work unit existed across the four classes at SP?
5. What are the differences, if any, in the ROI across the organizational
groups at SP?
Definition of Key Terms
Action Plan. A specific plan for the actions or steps that will be undertaken
to implement the ROI methodology within the organization. Action items focus on
specific spheres of influence. (Phillips, 2003)
Evaluation Framework. Defines the levels at which programs are
evaluated and how data were captured at different times from different sources.
The framework involves a four-level evaluation process: reaction, learning,
behavior, and results (Kirkpatrick, 1998).
Isolating Program Effects. Used to ensure accuracy in calculating the ROI
and to ensure an accurate picture of the program’s benefits. Excluding this step
in the process will result in an incorrect, invalid, and inappropriate ROI
14. 5
calculation.
Return on Investment (ROI). This is the ratio of earnings (net benefits) to
investments (costs); it is the most common measure for value-added benefits in
operational functions.
Brief Review of Related Literature
A review of literature related to the topic of return on investment in the
training industry revealed that there has been a body of knowledge generating
the standard approach to determining ROI. Several popular texts (Phillips, 1983;
Phillips, 1994; Phillips, 1997; Phillips, 2001) focus on the methodologies
presented by the ROI Institute (Phillips, 2002). Phillips, along with his partners
Patti Phillips and Ron Stone, created the ROI Institute as a means of
communicating the philosophies and methodologies of ROI to training and
human resource professionals around the world.
Other authors have considered the main views of Phillips and have made
some changes to the methodologies that were established. One major dissident
is Dennis Kravetz, with his own approach to measuring human capital. Kravetz’
approach accommodates the financial aspects of the complete human capital
concept, whereas Phillips’ approach focuses more on the business results
produced by an intervention—such as training (Kravetz, 2004). Although Kravetz’
approach is different, the results are generally complimentary to those used by
Phillips.
The American Society for Training and Development (ASTD) and the
International Society for Performance Improvement (ISPI) have published several
15. 6
books on the topic of ROI by different authors as well as three volumes of case
studies based on Jack Phillips’ ROI methodology (Phillips, 1994; Phillips, 1997;
Phillips, 2001). These case studies are the foundation of the methodology that
was used during the research. Because of the infancy of the methodology, there
is more potential for uncovering new insights and approaches to solving the
problem.
Highlights and Limitations of the Methodology
The research methodology used descriptive and inferential statistics to
characterize the data and to predict similarities. The primary analytical test used
was the one-way analysis of variance (ANOVA).
The data collection process involved using questionnaires and action
plans from the new or potential supervisors who participated in the training
program. It further included objectives and methodologies for each level of
evaluation targeted. These covered the following targets, by objective (Phillips,
1983; Kirkpatrick, 1998):
Reaction. Data collection included a participant feedback form at the end
of the training program to judge reactions to the training in regards to the
relevance and effectiveness.
Learning. Pre- and post-self-assessments, observed behaviors during skill
practice, and review activities were used to evaluate how much they learned.
Job Application. On-the-job behavior changes were monitored and
measured during the action plan implementation.
Business Impact. In the action plan, participants estimated the potential
16. 7
cost benefits of their applied behaviors over a 90-day period after training.
Questionnaires (Appendix A) were administered during a 90-minute
follow-up session which was scheduled about three months after the initial
training and co-facilitated by senior management. The business impact was
evaluated by comparing the identified measures on action plans (Appendix B) at
implementation of the high-performing leadership action plans (at the end of the
training cycle) during those 90 days following implementation.
Although several strategies were available to isolate the effects of training,
most of the methods were thought to be ill-suited to this situation. Participants’
direct estimates were found to be the most appropriate technique. Participants’
estimates of the impact of training have shown to be a reliable indicator of results
through practical application of this process by numerous ROI professionals
(Phillips, 1994, 1997, 2001). It is a proven methodology for deriving reliable data
and establishing reliable metrics and the best fit for use in this ROI study. Seven
examples of its use are provided in the Review of Related Literature.
Limitations of the Study
In the data collection, the focus was on impact and not process.
Consequently, very little effort was made to collect input on the actual training
delivery processes and mechanisms, themselves. Most of the emphasis was on
the effect of the program in relation to the investment required. To remain
objective, data were collected only from people who experienced the training.
A standard practice in ROI evaluation of short-term training programs is to
capture the first-year benefits after the program has been conducted (Brinkerhoff,
17. 8
1994; Graber, 1997). This practice, in essence, limits the analysis of benefits to
one year of operation. Although this could slightly overstate the results in some
cases, it represents a conservative. The benefits obtained in subsequent years
are not necessarily useful to the analysis. In this study, data were collected and
analyzed over a three-month period following the training. This data was then
extrapolated over a 12-month period to simulate standard practice.
It is recognized that not all data collected and analyzed is absolute and
that there may be qualifiers that need to be researched at a later date. There
may be variables that are qualitative in nature that are treated quantitatively for
the purpose of measuring results.
Research Expectations
The business impact of the training program was examined in this
quantitative study. Every attempt was made to uncover specific business results
linked to the training program. The impact of the training program was measured
by the extent of application of the skills and knowledge promoted in the program.
The program’s impact was indicated by the extent to which participants saw a
connection between the training program and the application of skills in the work
setting. It was further shown through their reactions’ consistency with their
responses on the action plans.
Intangible results are those benefits that cannot be assigned a dollar value
or those for which the assigned value is questionable. Even though these
benefits were not used in the ROI calculation, they are important to the goals of
SP. Comments from the learners’ action plan feedback indicated various
18. 9
intangibles that would benefit SP. There will be a follow-up study conducted by
SP to evaluate the intangible benefits.
19. 10
Chapter II: Review of Related Literature
After reviewing more than 150 journal articles and case studies, seven
case studies were selected as having the most relevance to the research topic.
Each of these articles deals with return on investment (ROI) in training programs
and is often cited by experts in the field. The following are short summaries of
each article and the different statistical approaches that each took.
How Much Is Performance Improvement Really Worth?
Berthiez (2001) conducted an ROI study on a sales training program for a
major global automobile corporation in Europe. The primary project objective
focused on the following questions: exactly what financial effect did this specific
training have on the overall bottom-line in sales of new cars? What percentage of
new sales, if any, could the training process claim to represent? The results were
substantial and unquestionably beneficial to executives in determining how to
allocate shrinking budgets to gain maximum return on human performance for
dollars invested (Berthiez, 2001).
The Phillip’s ROI methodology was used in this impact study. The steps
used were data collection, training effects isolation, data conversion to monetary
value, intangible benefits identification, program costs tabulation, and ROI
calculation (ROI methodology will be covered in Chapter 3). In addition to the
overall ROI model, it was found to be useful to add an additional component at
the beginning of the process model—the training needs analysis (TNA). The
focus on a TNA helped to identify clearly what needs to be accomplished with a
given training initiative (Berthiez, 2001).
20. 11
To isolate the relationship between training and performance
improvement, the following three approaches were used: training impact—sales
consultants’ perception of the influence of sales training on actual car sales;
confidence factor—sales consultants’ certainty of their estimates about the
influence of training and other factors; and customer validation—final sales data
collected by the customer and used to substantiate sale consultant estimates.
These approaches were selected for ease of use and the realistic credibility of
sources. Control groups, monitoring on-the-job application of principles learned in
training or trend-line analyses, could have been used to further isolate the data
on training effects. Berthiez suggests it would be beneficial to compare and
contrast other methods of isolating data in future ROI initiatives wherever
practical and cost efficient to do so (Berthiez, 2001).
The conversion of data was relatively easy, since units of cars sold can be
multiplied by a given unit price and unit margin to clearly establish the monetary
benefits. The data were calibrated and crosschecked against actual car sales
results reported in Standard-Poor’s annual report, objective industry statistics,
and internal company sales reports. The findings were discounted by a training
impact of 9% from data taken from the questionnaire responses. Training impact
represented sales consultants’ perceptions of the influence of the sales training
on actual car sales. The findings were discounted further by an average
confidence factor of 65%, representing sales consultants’ certainty of their
estimates regarding the influence of training and other factors. To maintain the
integrity of the statistical data, the study excluded any values that were outside of
21. 12
realistic possibility (Berthiez, 2001).
Data in this study was used to compare against known results of the
business and the industry. The results for the retail distributor—with an ROI of
325%–was conclusive evidence that the investors, the manufacturer, and the
retail distributor did realize a significant payback for the capital invested
(Berthiez, 2001).
Using ROI Forecasting to Develop a High-Impact, High-Value Training
Curriculum
With a variety of approaches to forecasting addressed, Graber (1997)
described the process used by a Midwest electrical power provider to allocate
funds for a variety of training initiatives and projects. The process built on the
principles of forecasting financial benefits and provided an important tool for the
training and human resource managers (Graber, 1997).
The purpose of the ROI forecasting was to identify the training that would
provide the highest possible payback and, more generally, to make wise training
and development decisions. The training itself was seen to have no inherent
value; the worth was dependent on the performance gains it catalyzed, the
performance gaps it addressed, and the opportunities it created in a given
environment. ROI forecasting did not affect the cost of training; however, it
maximized the payback from limited training resources and helped to avoid
training dollars going to waste (Graber, 1997).
The ROI forecasting process began by selecting employees and
supervisors who were most familiar with a job—the subject-matter experts
(SME). The SMEs agreed on the key accountabilities of the job, which were
22. 13
given a weight based on their importance and the typical time spent doing them
during a year. An estimation procedure (Casio-Ramos estimation) was used to
make the weights more accurate. Subject experts picked the highest weighted
accountability and gave it 100 points; every other accountability was then
compared to it and given a lesser number of points. Finally, the subject experts
identified from seven to ten critical skills for each key accountability (Graber,
1997).
Using a five-point rating scale (beginner, novice, skilled, advanced, and
expert), skill assessment questionnaires were completed separately by
employees and their managers, and both perspectives were weighed equally.
Employee skill gaps were identified and the cost of the gaps in terms of lost
performance was estimated. Rather than calculate the value of each employee,
the process was simplified by using the median of the employees’ pay range to
establish their value within each of three levels: professional, supervisory and
middle management (Graber, 1997).
To increase its value as a good measure of training need, the skill gap
was calculated differently than is typically done. A percentage skill gap value was
calculated using the traditional gap rating scale in conjunction with the
importance of the skill to the job. The dollar value of the job was then used to
calculate the dollar impact of the skill gap (Graber, 1997).
For example, if salary and benefits equaled $100,000, formal
presentations (the skill) make up a 4.8% weight, and the subject rated a 3 on
formal presentations (which equates to 50%), the calculation would be: (a)
23. 14
$100,000 x 4.8% = $4,800 (the skill value of a fully qualified employee), and (b)
$4,800 x 50% rating = $2,400 (the size of the gap from the optimum) (Graber,
1997).
Based on the skill gaps identified by this process for all applicable
employees, 11 training programs were chosen and the expected ROI for each
was calculated. These results show that only 6 of the 11 courses selected
showed a positive ROI; therefore, only about $11,800 was spent on this program
(Graber, 1997).
Measuring Return on Investment for a Mandatory Training Program
Marcial (2001) illustrated how a Florida-based government agency
measured the ROI for a mandatory training program on self-mastery. The
program evaluated the impact of using a specific training delivery methodology
and its ability to channel employees to participate in and contribute to the
organization (Marcial, 2001).
It involved a learning map on a high-performance development model
(HPDM), due to the perceived importance of self-mastery, and this particular
learning map had undergone several beta tests and revisions before it was used
by the agency. The learning map included workplace change sheets, how we
learn sheets, teammate skills sheets, development approach sheets, information
guides, and personal opportunity plans. Each person was asked to complete the
questionnaire alone, return to debrief the questionnaire as a group, identify one
thing he or she could contribute or do that could make a difference to the facility,
and tell two co-workers what he or she has learned about HPDM (Marcial, 2001).
24. 15
The data collection methodology was set up to take advantage of all the
data generated in the sessions. A comparison arrangement was established to
isolate the effects of the learning map. To isolate the effects of the learning map
further, the participants were asked to estimate the impact of the program
themselves (Marcial, 2001).
The researchers used the participants’ application of what they learned in
the learning map sessions to convert the data to a monetary value. Monetary
values were assigned to the changes made by the participants using regulations
and methods found within the agency. A database of employee time was readily
available from human resources. An internal specialist provided the
compensation data, including the cost of medical care for injury using billing
codes and allowances paid to providers for treatment (Marcial, 2001).
The benefits-to-cost ratio came out as 1.03, calculated by dividing the total
benefits ($2,819.37) by the total costs ($2,737.10). An ROI of 3% was found by
subtracting the total costs ($2,737.10) from the total benefits ($2,819.37) divided
by the total costs ($2,737.10) (Marcial, 2001).
At the project outset, an ROI of 25% was anticipated. Because this was a
mandated program, the whole cost was an expected expense with no financial
benefits; therefore the attempt to calculate the cost of mandatory training was
valuable. The use of the learning maps for HPDM did not appear to be an
economical delivery method at first, but it became evident that its use was worth
the time invested and that it had the potential to bring about a significant ROI
(Marcial, 2001).
25. 16
Resisting Measurement: Evaluating Soft Skills Training for Senior Police Officers
Police organizations are traditionally governed from the top down in a
military-like hierarchical structure. However, police work often requires the
exercise of independent judgment within limited contexts. McCarty’s (2001)
research illustrated the problems of implementing and evaluating a program
focused on interpersonal skills training in a highly structured, often resistant
organization in New York.
Two methods were used to collect data in this study, which was based on
a Dale Carnegie training program: action plans and questionnaires. Participants
used the action plans to track progress and to collect actual performance data
over a three-month period following the final training session. Participants
received the follow-up questionnaire three months after the final session so that
they could return it with the action plan. The questionnaire provided data
regarding the extent to which the participants had used the training on the job
and the results that came from these applications (McCarty, 2001).
Participants’ estimates of the impact of training were a reliable indicator
when appropriate steps were taken to collect data. Though their judgment was
subjective, the participants had direct experience to guide their estimates and
had first-hand knowledge of other influences that could have had an impact on
performance measures. Participants’ estimates had proven to be extremely
reliable in other studies where they were compared to results from control groups
(McCarty, 2001).
In McCarty’s study, the primary strategy for converting data to monetary
26. 17
value was to ask the participants to make estimates and calculations based on
improvements in their work units. On the action plans submitted, participants
used accepted standards and conversion factors to arrive at the monetary value.
Some of the action plans were incomplete or otherwise flawed, invalidating the
data for purposes of calculating the ROI; even then, there were indications of
performance improvement (McCarty, 2001).
To calculate the ROI, the benefits from the group were compared with the
fully loaded cost of the program for the group as follows: The total benefits were
$333,168 and the cost of the program was $136,530, therefore the ROI
($333,168 - $136,530 / $136,530) was 144%. The high yield for a small number
of contributors was indicative of the type of results manifested when senior
officials who have a large sphere of influence participate in action plan
improvements (McCarty, 2001).
A Preprogram ROI for Machine Operator Training
This proposed program included significant capital expenditures and the
creation of a Canadian training facility (Renaud, 1997). Prior to pursuing the
project, an ROI was developed using a small-scale pilot effort. The ROI was
developed using methods typically reserved for post-program evaluation. The
results of the process can apply to almost any type of setting in which a major
training expenditure is under consideration (Renaud, 1997).
According to Renaud, one of the most difficult tasks in completing this ROI
evaluation was estimating the expected benefits from the program. The pilot
program presented some measurable improvements and this information was
27. 18
used in five tangible benefit areas: training time, machining scrap, turnover,
safety, and maintenance expense (Renaud, 1997).
As a standard practice, supervisors recorded production shortfalls with
new employees until they reached the standard rate for a machine. These losses
were essentially production lost to trainees taking the time allowed to learn to
operate a machine at a standard rate. The pilot program showed a 64%
reduction in this production lost to training, and the supervisors estimated that
trainee losses could be reduced by 50% with a structured training program
(Renaud, 1997).
Many factors contribute to machining scrap; one of the biggest factors is
the lack of training of new and inexperienced operators. The supervisors
estimated that there could be at least a 10% reduction in total scrap costs with
the new training program. The turnover rate in the machining area was eight
employees per month; because of the smaller numbers of employees involved in
the pilot program, turnover reduction data were inconclusive. The supervisors felt
that training could reduce turnover by at least 30%; a 30% savings was
$115,200. This estimate was considered conservative (Renaud, 1997).
Most of the accidents in the machining area were not lost-time injuries.
The pilot program indicated a 25% reduction in accidents, but the supervisors
estimated that accidents could be reduced by 30%. To remain conservative, the
25% value was used, resulting in an annual savings of $14,250 (Renaud, 1997).
Effective training of new employees should result in less maintenance
required on production machines. The pilot program showed a dramatic
28. 19
reduction of 45%; however, the supervisors estimated that the unscheduled
maintenance expenses could be reduced by 10% each year with the
implementation of the training program (Renaud, 1997).
The total projected annual savings were $304,950. The annualized costs
were $131,500. The annual gross savings of $304,950 less the program costs of
$131,950 result in a net savings of $173,450. The expected ROI ($173,450 /
$131,500) for the first years was 132%. The investment in the equipment and the
program development was to be spread over several years (Renaud, 1997).
Getting Results with Interpersonal Skills Training
Because of their soft nature, interpersonal skills training was a particular
challenge when calculating ROI. Russ-Eft (1994) described a very successful,
commercially available interpersonal skills training program being implemented in
a large information service organization in New York.
The organization was facing several challenges during the implementation
of the training program. They had more than 100 locations spread across the
United States. They wanted to bring about a cultural change to make the climate
more supportive and cooperative and to foster improved performance throughout
the organization. The ROI and evaluation study was designed to identify
solutions to these cost-related issues (Russ-Eft, 1994).
The organization decided to use a financial approach to justify the
implementation of the skills program. The parameters were defined and the
evaluation was conducted using surveys designed to evaluate the transfer of
skills acquired. The evaluation instruments gathered ratings of subjects’ on-the-
29. 20
job behavior as well as ratings of organizational climate and job satisfaction. The
on-the-job behavior ratings included 44 items grouped into four categories:
dealing with problems, communicating with co-workers, working with supervisors,
and improving work. Members of the trained and control groups then indicated
the percentage of time that they spent on the job dealing with the items grouped
under the four behavioral categories. Ratings of organizational climate and job
satisfaction were obtained from seven additional items (Russ-Eft, 1994).
The Russ-Eft study used pre-training and post-training ratings of the
behavior of training and control groups, to which people had been randomly
assigned. Ratings were gathered from the members of both groups, their
supervisors, and their colleagues. Pre-training ratings were gathered immediately
before training; post-training ratings were gathered approximately three months
following training (Russ-Eft, 1994).
A series of analyses of variance with repeated measures were conducted.
These analyses compared trained participants with control participants, training-
group supervisors with control-group supervisors, and trained-group colleagues
with control-group colleagues (Russ-Eft, 1994).
The costs of training included trainees’ time away from work, trainers’ time
away from work for preparation and training, the costs of materials used during
the sessions, the time required for designing the sessions, and certification costs.
Costs incurred for the entire population of 85 trainees were estimated at
approximately $70,000 (Russ-Eft, 1994).
The analysis of variance showed overall improvement, retrospectively
30. 21
comparing skill ratings before and after training (i.e., post-training ratings of pre-
training skill) after the training was completed. Significant differences appeared
between trained and control groups due to these overall improvements (Russ-Eft,
1994).
The results indicated that the total benefits were approximately $305,000
for a sample of 42 trainees out of the total population of 85 trainees; resulting in a
net benefit of $235,000 when the costs are subtracted from the benefits. The
traditional ROI formula yields an ROI of 336%. These calculations underestimate
the net benefits somewhat, as none of the indirect benefits was included in the
analysis (Russ-Eft, 1994).
Training’s Contribution to a Major Change Initiative
Stone (1997) illustrated how the ROI was calculated for an extensive
training program for relationship bankers headquartered in North Carolina. The
program evaluated the impact of training in the face of a variety of other change
initiatives, including a process improvement effort implemented prior to the
training. The study illustrated one approach to isolate the impact of the various
factors contributing to improvement (Stone, 1997).
Because the decision to determine the ROI for training was not made until
after the program had been implemented, the process of calculating the ROI was
much more difficult. Several factors can contribute to performance
improvements. In the Stone study, the possible factors were the re-engineering
effort that preceded the training, the support of the deal team, incentives,
coaching by managers, capital market liaison assistance, and other training
31. 22
initiatives. Strategies were thus required to isolate the effects of the training. Four
strategies were considered: control group arrangement, trend-line analysis,
estimates taken directly from participants, and estimates taken directly from the
managers of the participants (Stone, 1997).
The data collection plan included several approaches to converting data to
a monetary value. The specific benefit from each of the six business performance
measures had to be converted to dollar values so they could be compared to the
training program costs. For the customer satisfaction measure, it was decided
that no value would be placed on the actual training; instead, customers would
be asked to indicate the specific benefit they received (Stone, 1997).
The participant and manager questionnaires provided significant
information on changes in behavior; the information obtained indicated the skills
were being used on the job. Based on the input from the team questionnaire, the
five business measures from the training program with the strongest influence
were increased sales of capital market products, improved customer satisfaction,
improved employee satisfaction, new business from existing clients, and new
relationships established. Customers were asked to provide specific information
regarding the impact of the training on the business; 67% of customers
responded that the bankers added value to their business (Stone, 1997).
The ROI for the training was developed only based on participants’ input.
The total fully-loaded cost to train the participants was $698,725. When this
amount is combined with the benefits in the standard ROI formula, the ROI
comes out to 47.2%. Although this value may be lower than anticipated, the
32. 23
return is much higher in reality; the 47.2% ROI value is an understatement of the
actual return that does not consider several factors. When factored in, the actual
ROI could easily approach a value in the 100% to 200% range (Stone, 1997).
ROI Case Studies
Table 1 has a list of additional case studies that are pertinent to using ROI
in a training environment and that hold relevance to this study.
Table 1
ROI Case Studies
Organization Industry Program ROI (%)
Office of Personnel U.S. Government Supervisory 150
Management Training
Magnavox Electronic Electronics Literacy Training 741
Systems Company
Litton Guidance and Avionics Self-Directed Work 650
Control Systems Team Training
Coca-Cola Bottling Soft Drinks Supervisory 1,447
Company of San Training
Antonio
Texas Instruments Electronics Sales Negotiation 2,827
Training
Apple Computer Computer Process 182
Manufacturing Improvement
Training
Hewlett-Packard Computer Sales Training 195
Company Support Services
First National Bank Financial Sales Training 555
Services
Causeway Corporation Financial Total Quality 154
Services Management
Training
33. 24
Table 1 (continued)
ROI Case Studies
Organization Industry Program ROI (%)
Multi-Marques Inc. Bakery Supervisory Work 215
Process Analysis
Training
Midwest Banking Banking Loan Officer 1,988
Company Training
Financial Services Inc. Financial Human Resource 2,140
Services Selection Training
North County Electric Electric and Gas Applied Behavior 400
& Gas Utility Management
Training
Yellow Freight System Trucking Performance 1,115
Management
Training
Healthcare, Inc. Healthcare Sexual Harassment 1,052
Services Training
Apex Corporation Manufacturing Advanced Sales 2,981
and Distribution Skills Training
Eastman Chemical Chemical Empowerment 2,307
Company Training
Nortel Learning Telecomm Finance Training 317
Institute
NYNEX Corporation Communications Information 511
and Media Technology
Training
Texas Instruments Technology Negotiation Skills 2,827
Systems Group Training
First Union National Banking Change Initiative 472
Bank Training
Bell Atlantic Network Telecomm Computer-Based 319
Services Maintenance
Training
34. 25
Table 1 (continued)
ROI Case Studies
Organization Industry Program ROI (%)
Speedy Telecomm Performance 1,600
Telecommunications Management
Company System Training
Cracker Box, Inc. Restaurant Chain Performance 298
Management
Training
Focus Corporation Computer Build-to-Customer- 570
Manufacturing Order Training
Verizon Communication Telecomm Customer Service -54
Skills Training
Slick Manufacturing Government Computer Training 125
Agency (Ireland)
Healthcare, Inc. Healthcare Sexual Harassment 1,052
Services Training
Apex Corporation Manufacturing Advanced Sales 2,981
and Distribution Skills Training
Eastman Chemical Chemical Empowerment 2,307
Company Training
Compiled from In Action: Measuring Return on Investment, Volume 1, by J. J.
Phillips (1994), In Action: Measuring Return on Investment, Volume 2, by J. J.
Phillips (1998), and In Action: Measuring Return on Investment, Volume 3, by J.
J. Phillips (2001).
35. 26
Chapter III: Methodology
Overview
The success of a training program at SP, a pharmaceutical company
located in the Midwest, was examined for this quantitative study. Two specific
objectives of this study were met through the implementation of a comprehensive
data collection and analysis process: (a) to examine the specific impact of the
training program in measurable business contributions to the extent possible, up
to and including the calculation of the ROI for SP; and (b) to examine the extent
to which participants applied on the job what they learned during the training.
At SP, the management had become more interested in measuring the
impact of training and development programs. Four major trends were driving
these actions:
1. Training programs were rapidly getting more expensive to develop and
deliver.
2. The importance of training in meeting strategic objectives within SP
placed the training process at a level where accountability was necessary.
3. A trend toward measurement and metrics at SP was recognized
because of regulatory compliance issues.
4. Executive management, in an attempt to manage resources efficiently
at SP, had brought closer scrutiny to the training and development process and
was requiring accountability for large training expenditures. Collectively, these
trends were driving a need for more accountability and evaluation in training and
employee development at SP.
36. 27
Restatement of the Problem
The problem addressed in this study was the determination if there were
positive financial impacts of a leadership-training program on a business and to
present a verifiable and valid ROI with meaningfulness. SP had a major need to
determine the value and applicability of its leadership training to the company for
current and future leaders. Therefore, a study of this nature was proposed and
was based on the hypotheses and research questions addressing the differences
in various learning relationships across four organizational groups in a targeted
company. There were many studies (Phillips, 1994; Phillips, 1997; Phillips, 2001)
that indicated a positive ROI, but the lack of a statistically sound approach in
noted studies (see Review of Related Literature for examples) hinders verifying
the reliability of the data in the studies.
Statement of Hypotheses
There are five hypotheses defined in this section. All five hypotheses each
involve an analysis of variance (ANOVA) test. These research questions were
developed to test the hypotheses to which they apply, not to prove them
Research Question 1: What differences, if any, in the perceived relevance
of the 15 objectives existed across the four classes at SP?
H10: There will be no difference in relation to the perception of relevance
of the 15 objectives across the four organizational groups at SP (Operations,
Sales and Marketing, Scientific Research, and Support Groups).
H1a: There will be a difference in relation to the perception of relevance of
the 15 objectives across the four organizational groups at SP (Operations, Sales
37. 28
and Marketing, Scientific Research, and Support Groups).
Research Question 2: What differences, if any, in the perceived relevance
of the six elements of the job existed across the four classes at SP?
H20: There will be no difference in relation to the perception of relevance
of the six elements of the job across the four organizational groups at SP
(Operations, Sales and Marketing, Scientific Research, and Support Groups).
H2a: There will be a difference in relation to the perception of relevance of
the six elements of the job across the four organizational groups at SP
(Operations, Sales and Marketing, Scientific Research, and Support Groups).
Research Question 3: What differences, if any, in the perceived relevance
of the five skill areas existed across the four classes at SP?
H30: There will be no difference in relation to the perception of relevance
of the five skill areas across the four organizational groups at SP (Operations,
Sales and Marketing, Scientific Research, and Support Groups).
H3a: There will be a difference in relation to the perception of relevance of
the five skill areas across the four organizational groups at SP (Operations, Sales
and Marketing, Scientific Research, and Support Groups).
Research Question 4: What differences, if any, in the perceived relevance
of the 13 topics in one’s own work or that of the work unit existed across the four
classes at SP?
H40: There will be no difference in relation to the perception of relevance
of the 13 topics in one’s own work or that of the work unit across the four
organizational groups at SP (Operations, Sales and Marketing, Scientific
38. 29
Research, and Support Groups).
H4a: There will be a difference in relation to the perception of relevance of
the 13 topics in one’s own work or that of the work unit across the four
organizational groups at SP (Operations, Sales and Marketing, Scientific
Research, and Support Groups).
Research Question 5: What are the differences, if any, in the ROI across
the organizational groups at SP? In this context, a significant difference simply
means there is statistical evidence that there is a consistent difference; it does
not mean the difference is necessarily large or important (Sleezer, 1994).
H50: There will be no meaningful ROI across the organizational groups at
SP (Operations, Sales and Marketing, Scientific Research, and Support Groups).
H5a: There will be meaningful ROI across the organizational groups at SP
(Operations, Sales and Marketing, Scientific Research, and Support Groups).
Description of Research Design
Phillips’ ROI Methodology: This ten-step model provides a process for
collecting data, summarizing and processing data, isolating the effects of
programs, converting the data to monetary value, and capturing the actual ROI.
In the first step, the planning is initiated and the specific business drivers
of the solution are identified. Discussion and decisions revolve around how the
solution will satisfy the business drivers. Business measures are clearly
identified. The objectives are established and revised to ensure that stakeholders
agree on the training to be applied, the behavior changes initiated and the
business impact measures to be influenced.
39. 30
Detailed planning takes place in the next step. The purpose of the
evaluation is clearly defined and baseline data is developed and collected. If the
purpose is to calculate the ROI, the entire ROI process (ten steps) will be
followed. This step includes determining the data collection strategy and
developing the necessary planning documents that specify in detail how steps
three through ten will be carried out.
The third step involves the client organization collecting the Level 1 and
Level 2 evaluation data during the solution implementation (the evaluation levels
are explained in Chapter 5).
In step four, data on the application of training, behavior changes and
business impact are collected. The business impact data is converted to
monetary values to calculate the ROI. Throughout the process, data is collected
at all levels to show a chain of impact up to the highest level that satisfies the
purposes of the study. Step five begins the data analysis phase of the process as
the effects of the solution are isolated to determine the extent that the business
measures were influenced by the solution.
The sixth step is only applied when the purpose of the evaluation includes
calculating the ROI. If stakeholders have determined that there is no interest in
the ROI calculation for a specific initiative, then the business impact and behavior
change data is reported minus the calculation.
Step seven reports data on intangible benefits along with business metric
improvements. Barriers and enablers to implementation and behavior change are
also reported. Any improvement in behavior and business metrics influenced by
40. 31
the solution is reported in step ten. Before the ROI is calculated (step nine), the
costs (step eight) are compared to the benefits that are converted to a monetary
value from step six. Additionally, all of the data from steps three, four, five, and
seven are reported, along with conclusions and recommendations. Conclusions
address information such as what caused the results, what worked, and what did
not work, while recommendations address where to go from there and how the
findings can be used to implement improvement.
In this study, two methods were used to collect data in keeping with the
Phillips’ ROI methodology. A follow-up questionnaire (see Appendix A) was used
to determine the extent to which participants utilized the training and achieved
self-reported on-the-job success. The action plan (see Appendix B) was
implemented during the training to identify areas for individual improvement as a
result of the training program, to link achievements to department-level
contributions, and to convert the contributions to monetary value. Participants
were required to continue using the process to track progress and collect actual
performance data for a three-month period after the last training session.
Participant estimates of training impact were a reliable indicator when
appropriate steps were taken to collect the data. The participants were the
closest individuals to the performance improvement and often were aware of the
other influences that affected the performance measures. For this study,
participants were asked to indicate the degree to which a specific improvement
resulted from the training program. The action plan was the tool used to capture
this data. The action plan was a tested and validated document approach
41. 32
devised by the ROI Institute (Phillips, 1997).
While data can be converted to monetary values in many ways, the
primary strategy that was used in this study was to ask participants to make
estimates and calculations based on the improvement in their work units. The
participants used accepted standards and conversion factors to arrive at
monetary values.
The leadership training program was believed to show positive business
results using the return on investment methodology. A structured approach was
employed, allowing a comparison of the data gathered from questionnaires and
action plans. The structure used in this study, designated by Gay and Airasian
(2000) as a quasi-experimental design, is well suited to situations where it is not
feasible to randomly assign individual participants to groups. This situation arose
from the purposive sampling methods that were applied in this study.
The following criteria were used for this quasi-experimental study: (a) four
training courses took place—a total of 48 employees were trained, (b) two
experienced senior leaders of SP, who were certified through an extensive train-
the-trainer program, conducted the class instruction, (c) the class was designed
for a group of 10-16 students, and (d) all assignments among all classes had the
same objectives.
Sample Size Selection. A population of 200 was identified by SP
management from which to select the sample size. It was determined that only
about 65 current employees of the leadership population of approximately 200
would require the training—based on a screening of performance reviews and
42. 33
succession planning data by SP, which were not a part of this study. Only 48
participants (24% of the total population, but 74% of the target population) would
be available for the training and subsequent study, therefore sample size
calculations were based on the following data. Using power analysis, the
acceptable margin of error was determined to be 5% with a confidence level of
95%. The sample size of 48 would yield a response distribution of less than 4%
based on an anticipated 100% response rate (which was achieved). With this in
mind, it would be likely to achieve meaningful input with a sample size of 48.
Therefore, with a sample size of 48 from the approximately 65 eligible and
required employees, a confidence level of 95% and a confidence interval of 7.32
(based on a sample size of 48 from a population of 65 and a worst-case
percentage of accuracy of 50%), about 73.8% of the target population made up
the sample of 48.
The size of the study sample was critical to producing meaningful results.
A power analysis could be used to determine the size of the sample is large
enough. However, with an unknown effect size or a useful standard deviation
(based on past data), a power analysis was not performed. All research
questions were tested to determine a meaningful impact in rejecting the null
hypothesis.
Operational Definition of Constructs and Key Variables
The independent and dependent variables for each research question are
presented in this chapter under the Statement of Hypotheses in the five research
questions and the accompanying null and alternate hypotheses. As well, the
43. 34
dependent variables for each research question can be found in the referenced
Statement of Hypotheses section.
Research Question 1. There were four organizational groups selected
(Operations, Sales and Marketing, Scientific Research, and Support Groups) with
each group being an independent variable. These four organizational groups
were used consistently throughout the research as the independent variable for
all six of the hypotheses and the associated research questions. Independent
variable data was collected using both the questionnaires and the action plans.
The averaged class score for the research question (based on the
questionnaire responses) was the dependent variable for the ANOVA test in
hypothesis one. For hypothesis one and research question one, the
questionnaire was the source of all data.
Research Question 2. There were four organizational groups selected
(Operations, Sales and Marketing, Scientific Research, and Support Groups) with
each group being an independent variable.
The averaged class score for the research question (based on the
questionnaire responses) was the dependent variable for the ANOVA tests in
hypothesis two. For hypothesis two and research question two, the questionnaire
was the source of all data.
Research Question 3. There were four organizational groups selected
(Operations, Sales and Marketing, Scientific Research, and Support Groups) with
each group being an independent variable.
The averaged class score for the research question (based on the
44. 35
questionnaire responses) was the dependent variable for the ANOVA tests in
hypothesis three. For hypothesis three and research question three, the
questionnaire was the source of all data.
Research Question 4. There were four organizational groups selected
(Operations, Sales and Marketing, Scientific Research, and Support Groups) with
each group being an independent variable.
The averaged class score for the research question (based on the
questionnaire responses) was the dependent variable for the ANOVA tests in
hypothesis four. For hypothesis four and research question four, the
questionnaire was the source of all data.
Research Question 5. There were four organizational groups selected
(Operations, Sales and Marketing, Scientific Research, and Support Groups) with
each group being an independent variable.
The averaged class score for the research question (based on the
questionnaire responses) was the dependent variable for the ANOVA tests in
hypothesis five. For hypothesis five and research question five, the action plans
were used as the data source.
Description of Materials and Instruments
Questionnaire. The participants received a questionnaire (see Appendix
A) that provided data regarding the extent to which participants used the training
on the job while involved in the program and the results that came from these
applications. The questionnaire had the participants: (a) rate the success of the
course in meeting 15 objectives, (b) rate the relevance of the program elements
45. 36
to the job, (c) indicate the degree to which the use of the 15 skills are enhanced,
and (d) indicate the extent to which one thinks this course will influence the
measures in their own work or that of the work unit. The questionnaire included a
13-item checklist, and it requested examples and details.
The questionnaire approach and format has been validated through
successful and effective use in multiple ROI projects and case studies completed
by the ROI Institute, as well as ROI practitioners worldwide (Philips, 1994;
Phillips, 1997; Phillips, 2001). Used in conjunction with the action plan, the
questionnaire was an invaluable assessment tool.
Action Plan: The requirement for the action plan (see Appendix B) was
communicated prior to the program start date. On the first day of training, the
program instructors described the action planning process in a 15-minute
discussion. The participants received notepads on which to capture specific
action items throughout the training program. They were instructed to make
notes when they learned a technique or skill that would be useful in improving
one of the three measures that they each identified as important. In essence, this
notepad became a rough draft of the action plans.
For this mixed method study, the reasons behind various aspects of the
results were relied on. In this study the why and how of improved leadership
were investigated. The need was for smaller and more focused samples rather
than large random samples. The researcher relied on qualitative methods for
gathering information on: (a) participation in the training, (b) direct experiential
learning and (c) analysis of outcomes through documentation. The actions plans
46. 37
provided qualitative data that was converted to quasi-quantitative data by the
participants.
The action planning process was discussed in detail in a one-hour session
during the second day of training. This discussion included three parts: (a) the
actual forms, (b) the guidelines for developing action plans and SMART (specific,
measurable, achievable, realistic, and time-based) requirements, and (c)
examples to illustrate what a complete action plan should look like.
The instructors distributed five blank action plans (only three are required,
one for each measure) and examples of completed action plans. During the
second day of training, the participants completed the booklets. The participants
worked in teams to complete all three action plans. Each plan took about 20-to-
30 minutes to complete.
During the third day of training, the participants briefly reviewed the action
planning process as a group, with each action plan taking about five minutes to
review. The program instructors then explained the follow-up steps to the group.
Selection of Subjects
Four functionally-defined groups (Operations, Sales and Marketing,
Scientific Research, and Support) were asked to submit the names of candidates
they considered key and potential leaders that would be contributing significantly
to the future of SP. They were considered the role models or leaders within their
respective departments, as determined by the executive leadership at SP. The
individuals that participated varied from new first-line supervisors to director-level
management.
47. 38
From a population of about 65 leaders companywide who were in need of
the leadership training, 48 were selected by executive management to participate
in the training. For the population of 48 participants, the breakdown by function
was Operations, 15; Sales and Marketing, 12; Scientific Research, 10; and
Support Groups, 11. The sample size of 48, with a confidence level of 95% and a
confidence interval of 7.32, allowed for a meaningful number of participants,
adding to the validity of the study.
Procedures
The training program was conducted in several sessions, with each
session lasting two to four hours and delivered one or two days a week for three
to four weeks. An orientation session also was conducted prior to the first
session. The training program included 48 participants from various functional
organizations of SP. Employees attended the training sessions on SP’s time.
This sample represented about 24% (48 participants) of a total target population
(about 65 targeted leaders and potential leaders) at SP.
The research included four phases: process planning, data collection, data
analysis, and the communication of results. This was summed up in an ROI
impact study of the Strategic Leadership training program within SP.
1. Process planning was the most critical phase. Thorough planning
ensured that the process addressed the appropriate objectives and used the
proper data collection instruments.
2. Data collection occurred at two periods—during the selected training
program(s) to measure participant reaction, satisfaction, and learning; and on a
48. 39
post-program basis to gather information on the application of skills and
knowledge as well as the impact the training program had on the organization.
3. The results of the program were shown by isolating the effects of the
training program. The costs were tabulated and the ROI calculation was
developed.
4. The communication of results included several issues that are often
neglected, but are as important as the process itself: (a) the process and
measurements were meaningless without communication; (b) communicating the
results was necessary to make improvements and to show accountability in the
training programs; (c) communication was a sensitive issue and could have been
a source of great benefit or a cause of major problems; and (d) the varieties of
target audience need different information.
Calculating the ROI required a value to be placed on each data element
connected with the training programs. The following are some strategies that
were used to convert data to monetary values.
1. Some output data converted to cost savings or profit contributions and
was able to be reported as a standard value, such as increased sales.
2. The cost of some quality measures were calculated and reported as a
standard value, such as customer complaints.
3. The historical costs of preventing a measure were used when
available, such as with time lost to accidents.
4. External databases contain an approximate value or cost of some data
elements, such as employee turnover.
49. 40
5. Internal and external estimates of the value of a measure, such as
employee complaints.
6. Measures linked to other measures for which costs are developed,
such as employee satisfaction linked to turnover.
7. Supervisors’ or managers’ estimates of costs or values, such as
unscheduled absence.
8. Employee time saved converted to wages and benefits.
9. Participants’ estimates of the cost or value of the data element, such
as work group conflict.
10. Training staff estimates of the value of a data element, such as
harassment complaints.
Converting data to monetary benefits was critical. The process was challenging,
but was methodically accomplished using one or more of the above strategies.
Data Collection Issues. A data collection strategy was designed to meet
the objectives of this study. The questionnaires and the action plans were utilized
to ensure that adequate, quality input was obtained for the evaluation. In both
data collection methods, the focus was on impact and not process.
Consequently, very little effort was made to collect input on the actual training
delivery processes and mechanisms themselves, although some data were
collected. Most of the emphasis was on the impact of the program, which was
obtained with evaluation Levels 3 and 4 for data collection and analysis (Phillips,
1983; Kirkpatrick, 1998).
To remain objective, data were collected only from people who took part in
50. 41
the training program. Although data from the instruction team and others could
have proven helpful, it was essential that input was free from any perceived bias.
These steps helped to ensure that the process was unbiased, objective, and
contained a minimum of errors.
Data Collection Timing. Another important issue to address in this study
was the timing of the data collection. Although the training program was designed
to have a long-term impact, the specific improvements from the training program
would be difficult to capture if assessed years after the training program was
completed. For longer periods, additional variables could influence output
measures, thus complicating the relationship between the training and the
improvement. The training program was time-spaced (one or two sessions a
week for four to six weeks), which provided opportunities for on the job
application of the training on an ongoing basis. Because of the above factors, it
was decided to measure the success of the training program during a three-
month period after the last training session. The data were collected and
analyzed over a three-month period following the training. This data was then
extrapolated over a 12-month period to simulate standard practice.
A standard practice in program evaluation is to capture the annual benefits
after the program has been conducted and compare them to the cost of the
program (Brinkerhoff, 1994; Graber, 1997). In essence, this limits the benefits in
an ROI calculation to the impact of a training program for one year of
improvements. While this could slightly overstate the results in some cases, it
usually understates them in practice. The skill transfer techniques used by the
51. 42
instructors were among the most effective in the training industry at building the
confidence and skills that contribute to long-lasting results.
Questionnaire and Action Plan. The most common follow-up method—
questionnaires—provided a rich source of information on the extent to which
participants applied what they had learned in the program and the success they
achieved with the application. Because of the need for business impact data, the
action plan process provided a capable means of gauging the actual impact of
the training program as its information was applied. The action plan was also a
useful tool to keep employees focused on changing their behavior in the work
setting. Appendix A is an example of the questionnaire and Appendix B is an
example of an action plan.
The action plan’s necessity was communicated prior to the program start
date. On the first day of training, the program instructors described the action
planning process in a 15-minute discussion.
The participants received the questionnaire 90 days after the completion
of the leadership-training program. It provided data regarding the extent to which
participants used the training on the job while the training program was ongoing
and the results that came from these applications.
Effects of Training Isolation. Participant estimates of training impact are a
reliable indicator when appropriate steps are taken to collect the data (Phillips,
2002). The participants were the closest individuals to the performance
improvement and were often aware of the other influences that affect the
performance measures. For this study, participants were asked to indicate the
52. 43
degree to which a specific improvement was caused by the training program. The
action plan was the tool used to capture this data.
Participants’ direct estimates were deemed the most appropriate
technique of evaluation for this study. Their estimates of the impact of training
are a reliable indicator of value (Phillips, 2002). The participants are the
individuals closest to the training and are often aware of the other influences that
have an impact on the leadership training measures. In studies where
participants’ estimates have been compared to the differences obtained from
control group experiments, their estimates were found to be very reliable
(Bernthal, 1994; McCarty, 2001; Russ-Eft, 1997; Westcott, 1994; Zigarmi, 1997).
Data to Monetary Value Conversion. While data could be converted to
monetary values in many ways, the primary strategy that was used in this study
was to ask participants to make estimates and calculations based on the
improvement in their individual work units. Participants used accepted standards
and conversion factors to arrive at monetary values.
Tabulating the costs of the training effort involved monitoring or
developing all of the costs related to the training program. A fully-loaded cost
profile was recommended when tabulating all of the direct and indirect costs
(Marelli, 1993). The return on investment was then calculated by comparing the
monetary benefits and costs. The benefit-cost ratio was obtained by dividing the
monetary benefits of the program by the costs. The return on investment used
the net benefits (costs minus benefits) divided by costs. This formula is
commonly used to evaluate other investments where the ROI is traditionally
53. 44
reported as earnings divided by investment.
Intangible Benefits. Intangible results are those benefits that cannot be
assigned a dollar value or the assigned value is questionable. Even though these
benefits were not used in the ROI calculation, they are important to the goals of
SP. Comments from the participants and action plan reporting indicated various
intangibles that will benefit SP.
Discussion of Data Processing
The data collection strategy was designed to meet the objectives of this
study. The action plans and the questionnaire were utilized to ensure that
adequate, quality input was obtained for the evaluation.
To assure anonymity, the questionnaire was sent confidentially from
Corporate Training and Development to the 48 participants and the participants
were not required to write their names on it. Experience has shown that
participants will provide more data that are valid if anonymous feedback is
ensured. They were under no pressure to exaggerate the data to impress
superiors. The questionnaires and action plans were returned to the Corporate
Training and Development department for review and analysis (note: the
researcher is also the Director of Corporate Training and Development). Then a
report was developed (an internal privileged document) and presented to the
Executive Leadership Council for presentation to executive management at SP
(see Appendix C).
Responses to the questionnaire provided a very good source of data
because of the number of write-in comments and the quality of data supplied.
54. 45
The return rate of completed questionnaires was 100%, and the return rate of
competed action plans was 100%. Descriptive and inferential statistics were used
to analyze the questionnaire and action plan data. The primary analytical test
used was the one-way analysis of variance (ANOVA). Tables and figures were
used to show the distribution of the participants’ selections in absolute numbers,
means and standard deviations.
Because the main thrust of this study was to determine the business
impact of the training program, every attempt was made to uncover specific
business results linked to the training program. The impact of the training
program was presented to indicate the extent of application of the skills and
knowledge obtained. Each participant was asked to select a number of skills they
used the most on the job since taking the training program.
The following tests were conducted in support of the five research
questions and five hypotheses.
Test 1. A one-way analysis of variance (ANOVA) was conducted to find if
there existed a difference in the perceived relevance of instruction across the
four organizational groups where the training took place. Each organizational
group was the independent variable with the averaged class score for question 1
on the questionnaire as the dependent variable.
Test 2. A one-way ANOVA was conducted to find if there existed a
difference in the perceived relevance of instruction across the elements of the job
for which the training took place. Each organizational group was the independent
variable with the averaged class score for question 2 on the questionnaire as the
55. 46
dependent variable.
Test 3. A one-way ANOVA was conducted to find if there existed a
difference in the perceived degree of enhancement from the instruction across
the skills of the job for which the training took place. Each organizational group
was the independent variable with the averaged class score for question 3 on the
questionnaire as the dependent variable.
Test 4. A one-way ANOVA was conducted to find if there existed a
difference in the perceived influence that instruction had on the measure of
performance in one’s own work or that of the work unit. Each organizational
group was the independent variable with the averaged class score for question 4
on the questionnaire as the dependent variable.
Test 5. A one-way ANOVA was conducted to examine if there existed a
difference in the ROI across the organizational groups. Each organizational
group was the independent variable and the ROI expressed as a percent was the
dependent variable.
Methodological Assumptions and Limitations
Assumptions. The underlying assumptions for this study were: (a) two
fulltime instructors were sufficient to deliver all the training throughout the entire
initiative, and (b) standard SP leadership training curricula, used previously within
SP, were suitable for use.
Limitations. SP was in charge of assigning who participated in the
leadership-training program; therefore, the researcher had no control over who
took part in the training and only limited control over class size and scheduling.
56. 47
Further, four organizational areas of one company were the focus of this study:
Operations, Sales and Marketing, Scientific Research, and Support Groups.
Therefore, the sample collected may not be representative of organizations in
other companies, limiting the ability to generalize.
Because the sample population obtained for this study was limited only to
key and potential leaders within SP, it is insufficient to use for generalizations
about the entire population of SP. The sample taken was representative only of
the population of key and potential leaders requiring leadership training at SP in
the four representative organizational groups, making it possible to generalize
about this population and these organizational groups alone.
Ethical Assurances
Of the many definitions of the term ethics, no one definition has emerged
as universally accepted. Any time ethics is the topic of discussion, terms such as
conscience, morality, legality, trust, values, responsibility, and integrity will
frequently be heard. Although these terms are closely associated with ethics,
they do not—by themselves—define it.
Assumptions about ethical underpinnings of human behavior are reflected
in the complexities involved in relating one culture to another, the distribution of
scarce resources, the allocation of power, the dynamics of groups, the
codification of ethical constructs, and the rewarding of ethical behavior and
discouraging of unethical behavior. As leadership becomes more complex and
deals with more situations, the application of ethics can also become more
complex (Bonhoeffer, 1995).
57. 48
Two standards are applied in order to help protect the privacy of research
participants. Almost all research guarantees the participants’ confidentiality; the
stricter standard is anonymity, which essentially means that the participant will
remain anonymous throughout the study. Clearly, the anonymity standard is a
stronger guarantee of privacy, but it is often difficult to accomplish—especially
when participants have to be measured at multiple time points (Blackburn, 1996).
Increasingly researchers have had to deal with the ethical issue of a
person’s right to service. Good research practice often requires the use of a non-
treatment control group. When the program may have beneficial effects,
however, persons assigned to the non-treatment control may feel their rights to
equal access to services are being curtailed (Blackburn, 1996).
Even when clear ethical standards and principles exist, there will be times
when the need to do accurate research runs up against the rights of potential
participants. No set of standards can possibly anticipate every ethical
circumstance. Furthermore, there needs to be a procedure that assures that
researchers will consider all relevant ethical issues in formulating research plans.
The participants of this study were employees of SP and agreed to
participate. They were broken into four groups, by organization, each of which
received the learning materials and participated in the training. Each person
completed the informed consent form (see Appendix D). The study was non-
invasive, and the participants were asked only to complete the questionnaire and
the action plan as appropriate for the group to which they were assigned.
With no risks to any participant, no additional safeguards needed to be
58. 49
established. All participants were identified by a randomly generated study
participant number that only the researcher had access to. All information
gathered was protected by confidentiality agreements and not shown to the
organization and other members of SP.
This researcher made every attempt to comply with the guidelines
established in Appendix B of the Dissertation Handbook (Northcentral, 2005).
Approval from the university’s Ethics Committee was granted via electronic
notification (see Appendix E).