1. WHEN PERFORMANCE IS FUZZY:
THE CRITICAL INCIDENT TECHNIQUE (CIT)
Steven W. Villachica, PhD
SteveVillachica@boisestate.edu
Organizational Performance
and Workplace Learning
Download slides and handout at
https://sites.google.com/a/boisestate.
edu/ieeci/e2r2p/project-deliverables

2. Agenda
2
 Introduction
 CIT Examples
 Your Turn
 Wrap Up

3. Where CPT Fits in Performance Improvement
3

4. What do you do when…
4
 Exemplary performance is fuzzy?
 No one knows what a “good one”
looks like?
 Managers and clients don‟t know how
work gets done?
 There are no functional descriptions of
workplace activities?
 Job descriptions  workplace tasks?
 What the organization says it values 
what the organization really values?

5. 5
Jonassen, Tessmer, Hoffman, Coffey, Ford, & Carnot (2001)
& Hannum (1999a,
1999b)
Harless (1986) Flanagan (1954, 1962)

6. CIT as Evidence-Based Practice
6
Data-Driven Decision Making Almost 60 Years of Use
1. Determine questions you  In peer-reviewed
want to answer journals
2. Collect specific, relevant data  In a variety of disciplines
from different, triangulated and workplace settings
sources in the field  To create all sorts of
3. Analyze the data performance
4. Draw conclusions to answer improvement solutions
the questions
--in ways that improve valued
performance
More stuff that works!

7. CIT in Many Fields
7
Researcher Used CIT to
Flanagan (1954) Create procedures to select and
train WWII aircrews.
Thomas & Identify triggers that virtual teams
Bostrom (2010) use to adapt their uses of
technology during a project
Korte (2010) Investigate how newly hired
engineers socialize themselves
within a firm

8. CIT in Many Fields
About 20 CIT appearances in PI and PIQ
8
Researcher Used CIT to
Craytor (1968) Create programmed instruction in
therapeutic radiology for nursing
students
Smith (2009) Identify areas of expertise
associated with ASTD‟s
competency model
Hale (2011) Create ISPI‟s proficiency-based
certification for school improvement
specialists

9. CIT in Many Fields
About 20 CIT appearances in PI and PIQ
9
Researcher Used CIT to
Lundberg, Conduct a needs assessment
Elderman, investigating a problem with billable
Ferrell, & hours in a national retailer‟s parts and
Harper (2010) service department
Bacdayan Create a test that quality improve-
(2002) ment teams can use to determine the
suitability of a given project
Dean (1998) How to conduct CIT
Marrelli (2005) How to conduct CIT

10. Handout
A Basic CIT Process pp. 1-2
10
4. Analyze data
1. Determine the aim
• Frame of reference
of the CIT
• Categories
• Priorities
2. Plan the CIT • Verification
• Observers
• Observations 5. Report findings
• Specific behaviors • Categories
• Prototypical incidents
3. Collect incidents • Limitations
• Observations • PI conclusions
• Interviews CIT isn’t a rigid recipe. It’s a set
• Focus groups of flexible set of principles.
• Surveys (Flanagan, 1954; Woolsey, 1986)

11. Example 1: Making Service Standards Real
11
 The opportunity
 The incidents
• Client (anonymous)
• Service Standards • Stephanie Clark
 Professional
• Amanda Collins
 Respectful
• Julie Kwan
 Compassionate
• Allison Sesnon
 Helpful
• What do the standards REALLY mean?
• How do we operationalize the standards?
• How do we close gaps in service performance?
 The results

12. What the ID Team Did
12
 Use CIT to collect stories about exemplary and
non-exemplary performance
 Generated competencies
 Ranked the criticality of the competencies
 Focused on two key competencies
• Responding to clientele needs
• Communicating with clientele and team
 Fixed the environment and provided training
• Standards, feedback, process

13. A Service Standard Example: Helpful
13
Exemplary Performance Non-Exemplary Behavior
An elderly guest, using a cane, came A family is in a hurry to get on the
into the kitchen for a yogurt. A road. They were just informed
volunteer working in the kitchen unexpectedly that they need to
greeted her and engaged in friendly check their child out of the
conversation. The volunteer hospital this afternoon. They are
recognized that the guest was frantically trying to get everything
having difficulty going out the door, done to leave. A volunteer notices
so the volunteer offered to hold the the family is leaving and reminds
door. The guest remembered she them to be sure to wash the room
needed a spoon for her yogurt, and laundry before they leave.
the volunteer fetched a plastic spoon
for the guest and assisted the guest
out the door.

14. How CIT Helped
14
 Made otherwise abstract standards
visible by associating compelling
stories with each
 Targeted service competencies
needing improvement
 Provided a mechanism to fix
environmental causes of the
performance gap
 Provided sources of demos and http://www.perform
ancexpress.org/2012
role-play activities for the training /08/tales-from-the-
 Made the effort “by and for the field-making-service-
standards-real-for-
volunteers and staff” families-in-need/

15. Example 2: Decreasing Time to Competent
Engineering Performance
15
 This material is based upon work
supported by the National Science
Foundation under Grant No. 1037808.
 Any opinions, findings, and conclusions
or recommendations expressed in this
material are those of the author(s) and
do not necessarily reflect the views of
the National Science Foundation.

16. Engineering Education Research to Practice
E2R2P
16
 Improve engineering education in ways that
improve workplace performance.
 Education engineering for engineering education.
Engineering Workplace
Newly Graduated and Hired
Students Skills
“Freshout” Engineers
Engineering Education Engineering Workplace
Research-to-Practice
Valley of Death

17. Our Shared Opportunity
Decrease Ramp Up Time to Competent Performance
17
Company Costs
Promotion! Desired $ Training
Competency
$ Errors
Actual
Competency $ Mentoring
Performance
$ Salary
New Task/Project
$ Opportunity
$ Other projects
Leave University/Enter Workforce
$ Others?
Time
{ }
Increase Starting Skills
- OR -
REDUCE Change Learning Curve
- OR -
CO$T Make Boundaries
Porous

18. Spanning Gaps between Actual and Desired
Engineering Performance
18
Shared Decrease Ramp-up Time to Competent Job
Education Engineering Opportunity Performance in the Engineering Workplace
Problem
Identification
Research Questions
• What are newly graduated and hired “fresh out” engineers
Root Escape doing/not doing in the workplace that they should?
Cause Cause • What are the consequences of performance/non-performance
Analysis Analysis in the workplace?
• What workplace competencies should fresh outs possess?
• In what workplace contexts do fresh outs apply the
competencies?
Escape • What are the root causes of workplace nonperformance?
Corrective
Corrective
Action
Action Focus Groups & Surveys
Problem Inspection • Engineering managers, engineering leads, HR personnel, and
Failures technical scientists who work with fresh out engineers
• Fresh out engineers  Engineering Practice Survey

19. Method
19
Participants
7 Focus Groups Company Manager Fresh-Out
 Qualitative design using critical Parametrix 5 0
Micron 4 3
incident technique (Flanagan, Motive Power 3 4
1954) CH2MHill 4 3
 16 engineering managers, lead Total 16 10
engineers, supervising
engineers, technical scientists,
and HR personnel that work
with freshouts to bring them up
to speed in the workplace
 10 freshouts

20. Method
Procedure
20
Identify Company Arrange Focus
Recruitment Sponsors Groups
Facilitate Collect Data
about Workplace
Collect Data
about Causes of
Focus Groups Performance Nonperformance
Grow Share Results Work towards
and Collaborative
Collaboration Sensemaking Corrective Action

21. Method Handout
Instrumentation p. 4
21
Critical Incident Card

22. Method
Instrumentation
13
Root Cause Analysis
INFORMATION TOOLS MOTIVATION
ENVIRONMENT
• Data • Resources • Incentives
• Expectations • Software • Rewards
• Feedback • Tools • Consequences
• Standard Operating • Support
Procedures
• Knowledge • Physical Capacity • Motives
PERSON
• Skills • Mental Capacity • Affect
• Flexibility • Work Habits
• Resilience • Drive

23. Comparing the Examples
23
 ID team supporting  Collect instances
nonprofit service  Group instances into
standards behaviors or competencies
 Research team  Prioritize the groups
conducting a  Make sense of the data
performance
analysis for “It’s the small stories
freshout engineers gathered together that
made that big ah-ah.”
 --Allison Sesnon

24. Your Turn!
Create and Classify Incidents for ISPI
24
ISPI wants to collect real-world stories
about performance improvement
1. Instances of successful HPT performance
2. Instances of unsuccessful HPT performance
3. Assign categories
4. Leave completed incidents on the table
You will need a pen or pencil!

25. Ground Rules
25
 Doing this voluntarily. Can stop any time.
 Feel free to omit your name and email if you
want.
 ISPI may contact you to learn more about an
incident you provide.
 ISPI considers your data confidential.
 ISPI will report data in aggregate form,
without mentioning individual contributors.

26. Name (optional) Email address (optional)
Describe an incident that occurred to a practitioner
(you or someone else) trying to improve
performance.
Discrete Tasks
26
• Deliver a client
What level of performance was the practitioner
trying to improve? presentation
(Check all that apply) • Identify the root
 Worker  Organization  Enterprise  Society cause of a problem
Does this incident reflect (check one): • Analyze a data set
 Where the practitioner successfully performed a
task related to improving performance?
 Where the practitioner was unsuccessful in • Keep stories short
performing a task related to improving
performance?
• Focus on quick
generation
What were the general circumstances leading up to
this incident?
Use Action Verbs
What the practitioner was trying to accomplish?
• Delayed production
How did this incident affect the goals of the
practitioner’s project?
• Increased costs
• Satisfied customers
How often do incidents like this occur?  Only once  Sometimes
• Met standards
 Frequently

27. Successful Incident (With apologies to Harold Stolovitch)
Describe an incident that occurred to a practitioner Describing a training request,
(you or someone else) trying to improve performance. the client seemed focused on
27
means (schedules, compliance,
length of training). Client didn’t
What level of performance was the practitioner trying
mention anything about the
to improve?
ends –the valued business
(Check all that apply)
goals that the training should
 Worker  Organization  Enterprise  Society
produce.
Does this incident reflect (check one):
 Where the practitioner successfully performed a Used probing questions to:
task related to improving performance? • Frame statements of actual
 Where the practitioner was unsuccessful in and desired performance.
performing a task related to improving • Align the gap with business
performance? goals.
What were the general circumstances leading up to Training request from human
this incident? resources department.
What the practitioner was trying to accomplish? Focus on valued performance
How did this incident affect the goals of the Refocused client on delivering a
practitioner’s project? valued success story
How often do incidents like this occur?  Only once  Sometimes
 Frequently

28. Unsuccessful Incidents
28
 Generate incidents where a practitioner was
unsuccessful in performing a task related to
improving performance.

29. Code the Incidents You’ve Created
Part 1
29
CPT Standard Code of Ethics Cause Analysis
(1-10) (A-F) (a-l)
For every instance:
1. Specify at least one
relevant CPT
standard (1-10)
2. Specify at least one
ethical code (A-F)

30. Code the Incidents You’ve Created
Part 2
30
CPT Standard Code of Ethics Cause Analysis
(1-10) (A-F) (a-l)
For unsuccessful
performances,
1. Indicate ONE
potential root
cause (a-l)
--Based on Gilbert (1978)

31. Initial E2R2P Findings
Problem Identification
31
What Freshouts Do on the Job—
Successfully and Otherwise
Communication and Teamwork
Design
12% Analysis
Technical fundamentals
Other, 23% Software skills
12% Problem solving
Motivation
2%
Positive attitude
2% Leadership
2% 9% Work Ethic
Circuit debug
2%
3% Trouble shooting and critical thinking
6% Real world engineering
3% 3%
3% Process Knowledge
6%
3% Programming
5% 5%
Business System Knowledge
Other

32. Initial E2R2P Findings
Root Cause Analysis
32
19% 17% 18% Env. Info
17%
4% Env. Tool
5%
0% 3% 0% Env. Mot
15%
Ind. Know
Ind. Cap
45% 57% Ind. Mot
Managers 6% Freshouts
8%
11% 35%
11%
29%
Dean (1997)

33. Next E2R2P Steps
33
 Outreach to professional
organizations, new company
sponsors, and other universities
 Present survey, problem
identification, and root cause
analysis findings to company
sponsors and participants for
collaborative sensemaking
 Create a community of shared
practice and concern https://sites.goog
le.com/a/boisest
 Build to a corrective action forum ate.edu/ieeci/e2r
with all stakeholders (a.k.a. “design
solutions”) 2p

34. CIT Wrap Up
34
 CIT has a track record spanning almost 60
years
 CIT is an evidence-based practice for
performance improvement
 CIT is applicable in a wide variety of settings
where performance is fuzzy
 CIT can be a valuable tool for
performance improvement
practitioners
When performance is
fuzzy, consider CIT!

35. References
35
Bacdayan, P. (2002). Preventing stalled quality improvement teams: A written test of project selection ability. Performance Improvement Quarterly, 15(1), 47-66. doi:
10.1111/j.1937-8327.2002.tb00240.x
Butterfield, L.D., Borgen, W.A., Amundson, N.E., & Maglio, A.-S.T. (2005). Fifty years of the critical incident technique: 1954-2004 and beyond. Qualitative
Research, 5(4), 475-497. doi: 10.1177/1468794105056924
Clark, S., Collins, A., Kwan, J., & Sesnon, A. (2012). Tales from the field: Making service standards real for families in need. Performance Xpress, (August 1).
http://www.performancexpress.org/2012/08/tales-from-the-field-making-service-standards-real-for-families-in-need/
Craytor, J.K. (1968). Critical incident technique, programmed instruction and nursing education. NSPI Journal, 7(6), 12-18. doi: 10.1002/pfi.4180070606
Dean, P.J. (1998). A qualitative method of assessment and analysis for changing the organizational culture. Performance Improvement, 37(2), 14-23. doi:
10.1002/pfi.4140370207
Flanagan, J.C. (1954). The critical incident technique. Psychological Bulletin, 51(4), 327-358. doi: 10.1037/h0061470
Flanagan, J.C. (1962). Measuring human performance. Pittsburgh, PA: The American Institute for Research.
Hale, J.A. (2011). Competencies for professionals in school improvement. Performance Improvement, 50(4), 10-17. doi: 10.1002/pfi.20208
Harless, J.H. (1986). Guiding performance with job aids. In M. Smith (Ed.), Introduction to performance technology (Vol. 1, pp. 106-124). Washington, DC: The National
Society for Performance and Instruction.
Hoffman, R.R., Coffey, J.W., Ford, K.M., & Carnot, M.J. (2001). Storm-lk: A human-centered knowledge model for weather forecasting. Paper presented at the Human
Factors and Ergonomics Society 45th Annual Meeting, Minneapolis/St. Paul, MN.
Jonassen, D.H., Tessmer, M., & Hannum, W.H. (1999a). Job task analysis. In Task analysis methods for instructional design (pp. 55-62). Mahwah, NJ: Lawrence Erlbaum
Associates.
Jonassen, D.H., Tessmer, M., & Hannum, W.H. (1999b). Procedural analysis. In Task analysis methods for instructional design (pp. 45-54). Mahwah, NJ: Lawrence Erlbaum
Associates.
Korte, R. (2010). „First, get to know them‟: A relational view of organizational socialization. Human Resource Development International, 13(1), 27 - 43. doi:
10.1080/13678861003588984
Lundberg, C., Elderman, J.L., Ferrell, P., & Harper, L. (2010). Data gathering and analysis for needs assessment: A case study. Performance Improvement, 49(8), 27-34. doi:
10.1002/pfi.20170
Marrelli, A.F. (2005). The performance technologist's toolbox: Critical incidents. Performance Improvement, 44(10), 40-44. doi: 10.1002/pfi.4140441009
Stone, D.L., Blomberg, S., & Villachica, S. (2009, April). Capturing and leveraging expert decision making and problem solving. Paper presented at the International Society
for Performance Improvement, Orlando, FL. http://www.dls.com/1175_CTA.pdf
Thomas, D.M., & Bostrom, R.P. (2010). Vital signs for virtual teams: An empirically developed trigger model for technology adaptation interventions. MIS
Quarterly, 34(1), 115-142.
Van Tiem, D.M., Moseley, J.L., & Dessinger, J.C. (2012). Performance improvement/HPT model--an overview. In Fundamentals of performance improvement: A guide to
improving people, process, and performance (3rd ed., pp. 41-59). San Francisco, CA: Pfeiffer. http://www.ispi.org/images/HPT-Model-2012.jpg
Woolsey, L.K. (1986). The critical incident technique: An innovative qualitative method of research. Canadian Journal of Counselling, 20(4), 242-254.

36. Thank You
36
Questions? Comments?
SteveVillachica@boisestate.edu