Ken Spero speaks about “Simulation Scorecards as Drivers for Calculating ROI in Addition to Feedback” at Serious Play Conference 2012 ABSTRACT: One of the biggest challenges facing learning professionals in organizations in every sector is justifying the cost of training. With the economy the way it has been, any level of spending is challenged with respect to the expected returns and without the justification, budgets are scarce. Much of the training that organizations need to achieve the necessary change in employees to affect performance relates to soft skills which are difficult to assess or measure under the best of circumstances. This is compounded by the fact that few training initiatives happen in a vacuum and it is therefore difficult to isolate the impact of the training versus other aspects of the initiative. For many years, Simulations have been a key vehicle for capturing and deploying Experience in a ‘risk free’ manner that approximates the ‘school of hard knocks’ but without the associated bruising. Articulating a Scorecard is a key step in Experience Design and Simulation development. Experience Design is built on the premise of being able to articulate and then operationalize the experience you want a participant to have. One of the initial key steps in articulating experience is to understand how success, or failure, is measured in that experience. By breaking down learning objectives to how they can be observed helps to make the intangible measurable. Once it is measurable, it is not so difficult to make it quantifiable.

1. Simulation Scorecards as Drivers for
Calculating ROI
in Addition to Feedback
Serious Play Conference Presented by:
August 22, 2012 Ken Spero

2. Today’s Session Outline
In this session we will address the following
issues relating to Scorecards and ROI:
1. Understanding the value of Simulation
2. The role of the Scorecard in Sim
3. Experience Design Methodology
4. Balanced Scorecard
5. Building the Business Case – ROI
6. Look at and Play a Sim (if we have time)

3. Why use Simulation?
Experiencing Best Practices Enables Critical Thinking

4. Why use Simulations?
Experience is the best teacher
 Sims provide experience (emotional engagement
with environment)
 Sims expand the evocable experience base, they
become part of your experience portfolio / “gut”.
 Sims consequate Mindlessness and encourage
Mindfulness
 Sims provides an opportunity for participants to learn
from failure, to Fail Forward
 Sims enable time acceleration to feel consequences
 Sim leverages the power of storytelling (context)

5. When Sim

6. Experience Design:
Authoring Simulations
(Introducing the Assignment)
1. Establish the Narrative (something that can be
re-used – Anchor Story)
a) Clearly define a problem / Target Learning Objective
b) Articulate the desired experience and outcome
c) Timeline and flow
d) Create Measurable Scorecard: Goals and Boundaries
e) Establish Context: Environment / Storyline / Cast
f) Start with ‘Golden Path’

7. Experience Design:
Authoring Simulations (Cont’d)
2. Articulate and Prioritize Key Learning Objectives
3. Write a Scene from narrative that reflects each of the
selected Learning Objectives
4. Articulate Alternatives/Choices/Feedback
5. Lay out scenes in the flow of the narrative
6. Determine Memorable and Realistic Consequences
7. Add intros and outros
8. Score Choices – Link to Objectives in a Measurable way

8. Simulation Scorecards
Scorecards are a very powerful component for
both:
Feedback
Authoring
Especially when there are tradeoffs between the
different objectives or between different
stakeholders

9. Example – Project Leadership
 Interpersonal and organizational aspects of projects
 Focus of Sim on both Team and other Stakeholders
 Team Scorecard Organizational Scorecard
– Focus - Customer Satisfaction
– Motivation - Management Satisfaction
– Skill Development - Perceived Product Quality
– Buy-in/Commitment - Team Total
– Productivity - Project Total
– Innovation - GANTT Chart (Scheduling)
– Communication - Budget
– Leadership

10. The Balanced Scorecard
Methodology
 Start with a business problem that needs to be
addressed – this is the underlying reason driving the
need for the training
 If we are able to solve this problem, what will I see
change or benefit or how does the problem we are
trying to fix manifest:
– From a Financial perspective
– With respect to my Customers and their Satisfaction
– With our Internal Management Processes
– With our ability to Innovate and Grow

11. Linking the Scorecard and ROI
 Back to the business problem …
 When it comes to ROI it is helpful to consider where
the potential returns will come from with respect to
solving the Business Problem:
– Quantified Business Benefits
– Avoided Costs and Reduced Risk
– Cost Changes

12. Developing the business case
Determine how and where the issues manifest at their
most compelling level by examining the:
Benefit – Beneficiary Matrix
– Benefit - What is the nature of the problem:
improved – Efficiency, Effectiveness or Innovation
– Faster, Better, New
– Beneficiary – Where is this most compelling place
to measure: Enterprise, Group or Individual

13. The Balanced Scorecard
Methodology
 Start with a business problem that needs to be
addressed – this is the underlying reason driving the
need for the training
 If we are able to solve this problem, what will I see
change or benefit or how does the problem we are
trying to fix manifest:
– From a Financial perspective
– With respect to my Customers and their Satisfaction
– With our Internal Management Processes
– With our ability to Innovate and Grow

14. Break Down Balanced Scorecard
to Measurable Components
 Identify observable behaviors that demonstrate the
scorecard items – these might be specific
competencies relating to the business problem or

15. Define Units of Measure
Identify units of Measure - from Benefit/Beneficiary Matrix
Individual Level
– Productivity – units/day
– Sales productivity – won/loss ratio
Group Level
– Sales org productivity – cost of sales
– Workgroup productivity - # of staff
– Improved Customer Service – number of complaints
Enterprise Level
– Improved Product Quality – defect rate
– New product capability – volume of new products/year
– New market entry – volume of new markets/year

16. Do the Math
 Use the units of measure to quantify benefits and/or
avoided costs and think about it over time
 Compile the expected cost of development and
deployment
 Establish a baseline cost – ‘what if we did nothing…’
 Also determine what you are competing against (i.e. an
alternative approach or some other usage of the
money)
 Find someone to help with the calculations, if you need
it, based on the expected complexity of the numbers
and metrics

17. For More Information
• Contact:
Ken Spero
(215) 565-5598
KSpero@NEXLEARN.com
www.ImmersiveLearningU.com

18. Useful Links:
www.seriousplayconference.com
www.seriousgamesdirectory.com
www.seriousgamesassociation.com
Contact:
sbohle@seriousgamesassociation.com