It has been estimated that up to 50% of training is wasted.* You can avoid this by monitoring how leaders are applying and benefitting from what they learn. What the webinar replay here: http://go.forum.com/NAMeasurementWebinarReplay

1. © 2013 The Forum Corporation.
Using Measurement to
Increase the Impact &
Effectiveness of Your Leadership
Development Programs
© 2012 The Forum Corporation.

2. 2© 2013 Forum Corporation
Your Presenters
Tom Atkinson, Ph.D.
Learning & Development Consultant
The Forum Corporation
Jeffrey Berk
Chief Operating Officer
KnowledgeAdvisors

3. 3© 2013 Forum Corporation
Why Measure Leadership Initiatives?
Improve
Prove
Reinforce

4. 4© 2013 Forum Corporation
You’ll walk away with:
1. Insights on the challenges and benefits of measuring
leadership development
2. The four big differences between the best
and the rest when it comes to measuring leadership development
3. Frameworks for developing a measurement plan
4. Case studies of what high-performing organizations are
doing to measure their leadership initiatives
Four Takeaways for Today

5. 5© 2013 Forum Corporation
Quiz
 How effective is your organization in using measurement to increase
the impact of your leadership development programs?
– We collect and use data to engage all stakeholders
– We collect and use data within the L&D function
– We collect data efficiently but don’t do much with it
– We collect data, but it’s a cumbersome process
– We have no formal measurement process

6. 6© 2013 Forum Corporation
Measuring and Demonstrating Impact:
What Top Performing Organizations Do
 They treat measurement as part of the solution —
just as a dashboard is part of a car and a lab test is part of a
medical exam.
 They involve all key stakeholders — especially business
sponsors — in devising the measurement plan.
 They scale the measurement process to the size of the
initiative and the need for precision (and err on the side of simpler
and less costly).
 They follow a disciplined process for collecting and
reviewing data to drive action — not to “see if anything
happened” but to “make it happen.”

7. 7© 2013 Forum Corporation
Gaining buy-in to
the initiative’s
goals, metrics, and
implementation
plan
Laying the
foundation for
behavior change
by teaching new
concepts and
skills
Supporting
leaders in
applying and
embedding new
attitudes and
behaviors on the
job
ALIGN EQUIP SUSTAIN
Measurement Helps Accelerate
Leadership Development
Executing, measuring and evaluating with discipline

8. 8© 2013 Forum Corporation
Actions:
Conduct a planning
meeting with
stakeholders
ALIGN EQUIP SUSTAIN
Elements of a Measurement Plan
Questions:
What are expected
outcomes?
Who is accountable for
what?
Actions:
Survey participants
and managers during
training
Questions:
Is the learning process
effective & efficient?
Are leaders prepared
to apply learning?
Actions:
Survey participants
and managers after
training; organizational
follow-up
Questions:
What is the evidence of
effectiveness?
What tools, training or
support are needed?

9. 9© 2013 Forum Corporation
Creating Alignment
What You
Measure
Examples of Specific
Metrics
L&D Accountability Line Accountability
1. Reaction
Increased awareness,
positive attitude toward
topics, motivation to
change
■ Course evaluation ratings
of learning satisfaction
■ Design quality
■ Develop materials
■ Select/ train instructor
■ Project management
■ Clarity of objectives
■ Subject matter experts and data
access
■ Visible support for training
2. Learning
Acquisition of concepts
and skills
■ Pre- and post-course
understanding of material
■ Measurement tools ■ Debrief learning with managers
■ Learning plans and meetings
3. Job Impact
Application of concepts
and skills on the job—
Behaviour change
■ 360-feedback scores
■ Employee satisfaction with
leadership behaviour
■ Observed employee
competencies
■ On-the-job application
activities
■ Application activity
debriefs
■ Model desired behavior
■ Post-training reinforcement
meetings
■ Observe/coach on specific
behaviours
■ Align learning priorities for
leaders with performance
management goals
4. Business Results
Achievement of
positive outcomes for
the company and/or
individual
■ Profitability and growth
■ Employee satisfaction
■ Turnover
■ Costs of recruiting
■ Design process to create
a management-led
measurement plan
■ Design & implement a regular
measurement plan
■ Manage processes, provide
technology, recognise & reward
behaviour change

10. 10© 2013 Forum Corporation
Creating Alignment
What You
Measure
Examples of Specific
Metrics
L&D Accountability Line Accountability
1. Reaction
Increased awareness,
positive attitude toward
topics, motivation to
change
■ Course evaluation ratings
of learning satisfaction
■ Design quality
■ Develop materials
■ Select/ train instructor
■ Project management
■ Clarity of objectives
■ Subject matter experts and data
access
■ Visible support for training
2. Learning
Acquisition of concepts
and skills
■ Pre- and post-course
understanding of material
■ Measurement tools ■ Debrief learning with managers
■ Learning plans and meetings
3. Job Impact
Application of concepts
and skills on the job—
Behaviour change
■ 360-feedback scores
■ Employee satisfaction with
leadership behaviour
■ Observed employee
competencies
■ On-the-job application
activities
■ Application activity
debriefs
■ Model desired behaviour
■ Post-training reinforcement
meetings
■ Observe/coach on specific
behaviours
■ Align learning priorities for
leaders with performance
management goals
4. Business Results
Achievement of
positive outcomes for
the company and/or
individual
■ Profitability and growth
■ Employee satisfaction
■ Turnover
■ Costs of recruiting
■ Design process to create
a management-led
measurement plan
■ Design & implement a regular
measurement plan
■ Manage processes, provide
technology, recognise & reward
behaviour change

11. 11© 2013 Forum Corporation
Creating Alignment
What You
Measure
Examples of Specific
Metrics
L&D Accountability Line Accountability
1. Reaction
Increased awareness,
positive attitude toward
topics, motivation to
change
■ Course evaluation ratings
of learning satisfaction
■ Design quality
■ Develop materials
■ Select/ train instructor
■ Project management
■ Clarity of objectives
■ Subject matter experts and data
access
■ Visible support for training
2. Learning
Acquisition of concepts
and skills
■ Pre- and post-course
understanding of material
■ Measurement tools ■ Debrief learning with managers
■ Learning plans and meetings
3. Job Impact
Application of concepts
and skills on the job—
Behavior change
■ 360-feedback scores
■ Employee satisfaction with
leadership behavior
■ Observed employee
competencies
■ On-the-job application
activities
■ Application activity
debriefs
■ Model desired behavior
■ Post-training reinforcement
meetings
■ Observe/coach on specific
behaviors
■ Align learning priorities for
leaders with performance
management goals
4. Business Results
Achievement of
positive outcomes for
the company and/or
individual
■ Profitability and growth
■ Employee satisfaction
■ Turnover
■ Costs of recruiting
■ Design process to create
a management-led
measurement plan
■ Design & implement a regular
measurement plan
■ Manage processes, provide
technology, recognise & reward
behaviour change

12. 12© 2013 Forum Corporation
Creating Alignment
What You
Measure
Examples of Specific
Metrics
L&D Accountability Line Accountability
1. Reaction
Increased awareness,
positive attitude toward
topics, motivation to
change
■ Course evaluation ratings
of learning satisfaction
■ Design quality
■ Develop materials
■ Select/ train instructor
■ Project management
■ Clarity of objectives
■ Subject matter experts and data
access
■ Visible support for training
2. Learning
Acquisition of concepts
and skills
■ Pre- and post-course
understanding of material
■ Measurement tools ■ Debrief learning with managers
■ Learning plans and meetings
3. Job Impact
Application of concepts
and skills on the job—
Behavior change
■ 360-feedback scores
■ Employee satisfaction with
leadership behavior
■ Observed employee
competencies
■ On-the-job application
activities
■ Application activity
debriefs
■ Model desired behavior
■ Post-training reinforcement
meetings
■ Observe/coach on specific
behaviors
■ Align learning priorities for
leaders with performance
management goals
4. Business Results
Achievement of
positive outcomes for
the company and/or
individual
■ Profitability and growth
■ Employee satisfaction
■ Turnover
■ Costs of recruiting
■ Design process to create
a management-led
measurement plan
■ Design & implement a regular
measurement plan
■ Manage processes, provide
technology, recognize & reward
behavior change

13. 13© 2013 Forum Corporation
KnowledgeAdvisors
Jeffrey Berk
Chief Operating Officer
KnowledgeAdvisors

14. 14© 2013 Forum Corporation
Leadership program Key Tools
June 26, 2013 14
1. Identify Leadership KPIs
2. Create Leadership ‘Smart Sheet’
3. Design the Leadership Dashboard
4. Provide a Leadership Program Executive Report
5. Deploy On-the-Job Reinforcement
6. Conduct Business Impact Exercise

15. 15© 2013 Forum Corporation
Identify Leadership KPIs
15
Outcome Metrics - Business Outcomes
Leader Retention
Employee Engagement
Employee Retention
Leader Performance Ratings
Leader Financial Management
Effectiveness Metrics – Program
Performance
Delivery Quality
Knowledge Gain
Application to Job
Business Alignment
Value
Efficiency Metrics – Cost and Activity
# of Participants
% of Employees Trained
Total L&D Investment
L&D Investment per Participant
Courses Utilized
Classes Conducted
Instructors Utilized
Locations Used

16. 16© 2013 Forum Corporation
Create Leadership ‘Smart Sheet’
16
Not only
evaluate
The training
but also the
process
such as on-
boarding or
leadership
development
or sales
effectiveness
.

17. 17© 2013 Forum Corporation
Smart Sheet Forecasting
Connecting training to specific outcomes

18. 18© 2013 Forum Corporation
Design Leadership Dashboard
18
 A visual, concise data
representation
 Shows the efficiency,
effectiveness and outcome
metrics
 Data from evaluations, tests,
assessments imports, LMS /
HRIS feeds, uploads, etc.

19. 19© 2013 Forum Corporation
Leadership Executive Reporting
19
 Reporting for a non L&D
audience that emphasize
efficiency, effectiveness
and outcomes.
 Reporting for L&D
executives that focus on
satisfaction to impact and
value
 Hold quarterly reviews
with L&D execs and
sponsors

20. 20© 2013 Forum Corporation
On-the-Job Reinforcement
20
Use goal setting tools to establish and track changes in business results or
behaviors when participants are on-the-job. It also can identify scrap learning.

21. 21© 2013 Forum Corporation
Measuring Leadership
Development Programs:
Case Examples

22. 22© 2013 Forum Corporation
Manager Engagement Case Example
 Consumer goods strategic program
 Two teams with 2 different managers trained
 One manager used reinforcement post training
 One manager didn’t use reinforcement
 Manager w/ reinforcement had 67% change
 Manager w/out reinforcement had 33% change
22June 26, 2013© 2012,
KnowledgeAdvisors

23. 23© 2013 Forum Corporation
Leadership Program Impact Example
23
 Leadership program with optional ‘project’
 Determine if optional project influenced employee retention in
leadership program
 Employees with no learning projects averaged a turnover rate of
21%
 Employees with 2 or more learning projects averaged a turnover
rate of 2%
 Clearly the control group vs. learning group shows a difference
Source: Dr. John Sullivan, Presentation at Analytics Symposium

24. 24© 2013 Forum Corporation
 Business context
– Global energy company
– Large-scale merger
– Need to drive operational efficiencies and results
– Shift to leaner, team-oriented environment
 Implications for front-line supervisors
– Driving functional results and meeting organizational objectives
– Developing employees to be productive and effective
– Achieving results through others
– Developing future leaders
Front-line Leadership Example

25. 25© 2013 Forum Corporation
Front-line Leadership
Core Program Elements
Measurement and Documentation:
 MTM “Smart Sheet” Surveys for Level 1 (Reaction), Level 2 (Learning), and
Level 3 and 4 (Subjective Evaluation of Impact and Results)
 MTM Follow Up Surveys with Participants and Managers for Level 3 and 4
(Impact and Results)
Intermodule
Launch
Webinar:
Setting You Up
for Success as
a Supervisor
4-day Classroom Program:
Specific to Front-line manager
competencies such as coaching,
getting results through others, etc.
Reconnect
Webinar:
Share and
Evaluate
Results,
Recommit
Intermodule

26. 26© 2013 Forum Corporation
Front-line Leadership Program
Measurement Reporting
• Utilization Reporting
• Cost Reporting
Efficiency Measures:
Measures the extent to which
time or effort is well used for the
intended task or purpose
• Metrics the Matter Program Reporting
• Post Report
• Follow-up Report
Effectiveness Measures:
Measure the quality of the
program
• Selected Business Metrics
Outcome Measures:
Measures the relationship
and/or value of the program to
broader business goals

27. 27© 2013 Forum Corporation
Leadership Development
Measurement Diagnostic
Think of an upcoming Leadership Development Initiative (LDI) you are leading (e.g. high potential leadership
development system, onboarding for supervisors, management essentials, leading change program). Answer
the following questions with respect to that LDI, indicating low, medium or high.
To What Extent… Low Med. High
1. Are executives involved in shaping/defining this specific LDI?
2. Does this LDI link to key business metrics?
3. Will leadership skills/tools introduced in this LDI be immediately applicable on the job for
participants?
4. Will executives of the company be directly involved in implementing this LDI?
5. Is there a measurement discipline and cadence for reporting on talent management?
6. Do managers of participants in this LDI hold their direct reports accountable for critical
leadership behaviors?
7. Is there a measurement discipline and cadence for reporting on leadership development?
8. Is there a documented and communicated leadership development strategy aligned to the
broader business strategy?
9. Is real work application (e.g. individual or group action learning) a part of this LDI?
10.Do managers of participants in this LDI provide ongoing development opportunities, support and
coaching to their direct reports?
TOTAL
(Add up your checks in each column for use on the next page)

28. 28© 2013 Forum Corporation
Measurement Diagnostic Results
Leadership Development Strategy
Total the number of your “low,”
“medium” and “high” ticks from the
Leadership Development
Measurement Diagnostic.
Mostly“Low”
Mixof
“Medium”and
“Low”
Mixof”
Medium”and
“High”
Mostly“High”
Efficiency measures: Measures the extent to which time or effort is
well used for the intended task or purpose
   
Effectiveness measures: Measure the quality of the program from
the participant perspective immediately following the program
   
Effectiveness measures: Measure the quality of the program from
the participant/participant manager perspective with follow up
reports (60 days after the program)   
Outcome measures: Measures the relationship and/or value of the
program to broader business goals using existing business KPIs

Define what you will measure (below) based on
your responses (at right)

29. 29© 2013 Forum Corporation
Four Takeaways for Today
We have shared:
1. Insights on the challenges and benefits of measuring
leadership development
2. The four big differences between the best
and the rest when it comes to measuring leadership development
3. Frameworks for developing a measurement plan
4. Case studies of what high performing organizations
are doing to measure their leadership initiatives

30. 30© 2013 Forum Corporation
At Forum, we help our clients develop their leaders, managers and sales
teams to realize specific business objectives faster and more effectively. We
are proud to provide award-winning, comprehensive training solutions backed
by in-depth research to clients throughout North America and around the
world.
marketing@forum.com
forum.com
KnowledgeAdvisors is a talent and learning analytics solutions
and technology firm that helps organizations measure,
communicate and improve the impact of their people by better
managing processes through reliable metrics. Its proprietary
measurement systems and benchmarking expertise help
companies more successfully manage their learning and talent
management investments.
info@knowledgeadvisors.com knowledgeadvisors.com
Questions?