1. Measuring Strategic, Visible and Costly
Learning Programs
Speaker: Jeffrey Berk
Chief Operating Officer
KnowledgeAdvisors, Inc.
Moderator: Kellye Whitney
Managing Editor
Chief Learning Officer magazine
#CLOwebinar

2. Tools You Can Use
• Q&A
– Click on the Q&A panel (?)
in the bottom right corner
– Type in your question in the
space provided
– Click on “Send.”
#CLOwebinar

3. Tools You Can Use
• Polling
– The poll will appear on the
right side of your screen
– Select the best option for
each question
– Click on “Submit”
#CLOwebinar

4. Frequently Asked Questions
Will I receive a copy of the webinar recording?
YES
Will I receive a copy of the slides?
YES
Please allow up to 2 business days to receive these materials
#CLOwebinar

5. Measuring Strategic, Visible and Costly
Learning Programs
Kellye Whitney
Managing Editor
Chief Learning Officer magazine
#CLOwebinar

6. Measuring Strategic, Visible and Costly
Learning Programs
Jeffrey Berk
Chief Operating Officer
KnowledgeAdvisors, Inc.
#CLOwebinar

7. Measuring Strategic, Visible and Costly Learning Programs
Jeffrey Berk
Chief Operating Officer

8. Agenda
§ Review the importance of measuring strategic, visible and
costly programs
§ Discuss the measurement plan to gather timely, credible
data
§ Showcase sample reports, statements, dashboards and
scorecards
November 29, 2011 © 2011, KnowledgeAdvisors 8

9. About KnowledgeAdvisors
November 29, 2011 © 2011, KnowledgeAdvisors 9

10. Some of Our Clients

11. A Program with an Unhappy Ending
1. Company A went back 2 years… and identified
that they had 1,019 people rated #3, “needs
improvement” in recent performance reviews
2. They calculated that they spent an average of
$13,090 per year on highly visible performance
management (Coaching, counseling,
mentoring, retraining and manager time) to
elevate them to #1’s ,“high performers”
3. They identified 2 years later… the number that
were currently ranked as #1’s, was ZERO
Source: Dr. John Sullivan, Presentation at Analytics Symposium
November 29, 2011 © 2011, KnowledgeAdvisors 11

12. When Do I Need to Measure?
For any L&D program, ask the following questions:
ü Is it Strategic?...aligned to an executive initiative
ü Is it Visible?...awareness by senior management
ü Is it Costly?...more money than a typical
program
strategic costly
visible
November 29, 2011 © 2011, KnowledgeAdvisors 12

13. Question?
What Percent of Your Programs are Strategic,
Visible and Costly?
A. 0%
B. <5%
C. 6 to 10%
D. 11 to 25%
E. >25%
November 29, 2011 © 2011, KnowledgeAdvisors 13

14. Strategic, Visible, Costly Programs
§ Leadership Development
§ On-boarding/ New Hire
§ Sales Training
§ Others?
November 29, 2011 © 2011, KnowledgeAdvisors 14

15. Desired Measurement Outcomes
§ Quantified feedback on program satisfaction and quality
§ Timely indicators of impact and effectiveness
§ Provide stewardship and accountability to stakeholders
§ Implement world-class, credible and scalable program
measurement techniques
§ Access useful data-driven information to prioritize
continuous improvements to the program
§ Create a sustainable measurement process to measure
the program long-term and to easily utilize on other
learning programs
November 29, 2011 © 2011, KnowledgeAdvisors 15

16. Question?
Do you currently measure the business impact of
strategic, visible and costly programs?
A. Yes
B. No
November 29, 2011 © 2011, KnowledgeAdvisors 16

17. Measurement Plan ‘Top 10’
1. Identify Key Performance Indicators
2. Create Smart Sheet Evaluations
3. Build Communication Plan
4. Design the Dashboard
5. Provide Executive Reports, Scorecards, and Statements
6. Conduct Capability Assessment
7. Perform a Test/Certification
8. Gather Sponsor Satisfaction Data
9. Deploy On-the-Job Reinforcement
10. Conduct Business Impact Exercise
November 29, 2011 © 2011, KnowledgeAdvisors 17

18. Identify Key Performance Indicators
Outcome Metrics - Business Outcomes
• Leader Retention (Leadership)
• Time to Performance (On-boarding)
• Win Rate (Sales)
Effectiveness Metrics – Program
Performance
• Delivery Quality
• Knowledge Gain
• Application to Job
• Business Alignment
• Value
Efficiency Metrics – Cost and Activity
• # of Participants
• % of Employees Trained
• Total L&D Investment
• L&D Investment per Participant
• Courses Utilized
• Classes Conducted
• Instructors Utilized
• Locations Used
November 29, 2011 © 2011, KnowledgeAdvisors 18

19. Create ‘Smart Sheet Evaluations’
Not only
evaluate
The training
but also the
process
such as on-
boarding or
leadership
development
or sales
effectiveness
.
November 29, 2011 © 2011, KnowledgeAdvisors 19

20. Build Communication Plan
A communication plan is a document that
can be distributed to participants and
sponsors that outline the context for
measuring the program as well as what,
when and how measurement will be done.
This helps with change management and
project management.
November 29, 2011 © 2011, KnowledgeAdvisors 20

21. Design the Dashboard
§ A visual, concise data
representation
§ Shows the efficiency,
effectiveness and
outcome metrics
§ Data from evaluations,
tests, assessments
imports, LMS / HRIS
feeds, uploads, etc.
November 29, 2011 © 2011, KnowledgeAdvisors 21

22. Reports, Scorecards, Statements
§ Reporting for a non L&D
audience that emphasize
efficiency, effectiveness and
outcomes.
§ Reporting for L&D
executives that focus on
satisfaction to impact and
value
§ Hold quarterly reviews with
L&D execs and sponsors
November 29, 2011 © 2011, KnowledgeAdvisors 22

23. Conduct Capability Assessment
Conduct a 180 or 360 of core competencies to identify blind spots and hidden
strengths specific to the desired behaviors participants are to improve
November 29, 2011 © 2011, KnowledgeAdvisors 23

24. Perform Test/Certification
Validate critical knowledge and increase participant confidence with a test.
1. Take Test
2. Receive Feedback
3. Receive Certificate
November 29, 2011
November 29, 2011 © 2011, KnowledgeAdvisors
© 2011, KnowledgeAdvisors 24

25. Sponsor Satisfaction Data
Gather feedback from key sponsors at key milestones regarding their overall
satisfaction with the program and its outcomes.
November 29, 2011 © 2011, KnowledgeAdvisors 25

26. On-the-Job Reinforcement
Use goal setting tools to establish and track changes in business results or
behaviors when participants are on-the-job. It also can identify scrap learning.
November 29, 2011 © 2011, KnowledgeAdvisors 26

27. Business Impact Exercise
§ Control Group (side by side)
§ Determine if learning projects influenced
employee retention
§ Employees with no learning projects
averaged a turnover rate of 21%
§ Employees with 2 or more learning
projects averaged a turnover rate of 2%
§ Clearly the control group vs. learning
group shows a difference
Source: Dr. John Sullivan, Presentation at Analytics Symposium
November 29, 2011 © 2011, KnowledgeAdvisors 27

28. Question?
Do you feel your measurement process for
measuring strategic, visible and costly programs
is practical, credible, and repeatable?
A. Yes
B. No
November 29, 2011 © 2011, KnowledgeAdvisors 28

29. Measurement Success – Happy Ending
Executive Program for Government Entity:
An executive leadership program generated more
than double the impact of typical leadership programs
equating to nearly 70% on-the-job application and a 2
to 1 benefit to cost ratio.
Strategic Sales Training
A strategic sales training program generated $74,000
in increased sales per sales person for a $5,000 per
person investment. This data was the catalyst that
convinced management to expand the program to
other sales territories.
Front-Line Staff New Hire Program
An on-boarding program for front-line staff brought
an additional 15,000 hours of increased productivity to
the business as a result of the increased speed to
competency. This analysis helped save a new hire
program that was questioned by management.
November 29, 2011 © 2011, KnowledgeAdvisors 29

30. Contact Information
For further information on the information contact:
Jeffrey Berk
Chief Operating Officer
+1 312 676-4411
jberk@knowledgeadvisors.com
www.knowledgeadvisors.com

31. Join Our Next CLO Webinar
Developing a Mobile Learning Strategy
Thursday, December 8, 2011
CLO Webinars start at 2 p.m. Eastern / 11 a.m. Pacific
Register at www.clomedia.com/events
Join the CLO Network: http://network.clomedia.com/
#CLOwebinar