1.
What the NTSB teaches us about
incident management & postmortems
Michael Kehoe
Staff Site Reliability Engineer

2.
Agenda and Vision

3.
Today’s
agenda
1 Introductions
2 Background on the NTSB
3 NTSB: Investigative Process
4 Recommendations & Most Wanted List
5 How this applies to us?
6 Final thoughts

4.
Michael Kehoe
$ WHOAMI
• Staff Site Reliability Engineer @ LinkedIn
• Production-SRE Team;
• Disaster Recovery
• Incident Response
• Visibility Engineering
• Reliability Principles
• Find me online at:
• @matrixtek
• https://michael-kehoe.io
• linkedin.com/in/michaelkkehoe

5.
Production-SRE Team @ LinkedIn
$ /USR/BIN/WHOAMI
● Disaster Recovery - Planning & Automation
● Incident Response – Process & Automation
● Visibility Engineering – Making use of
operational data
● Reliability Principles – Defining best practice
& automating it

6.
Incident Command System (ICS)
https://training.fema.gov/emiweb/is/icsresource/assets/reviewmaterials.pdf

7.
Background on the NTSB

8.
Background on the NTSB
JURISDICTION
● Aviation
● Surface Transportation
● Marine
● Pipeline
● Assistance to other agencies/ governments

9.
“The NTSB shall investigate or have investigated and
establish the facts, circumstances, and cause or
probable cause of accidents…”
U.S. Code § 1131

10.
“… The Board shall report on the facts and
circumstances of each accident investigated…The
Board shall make each report available to the public
at reasonable cost…”
U.S. Code § 1131

11.
“The NTSB does not assign fault or blame for an
accident or incident…accident/incident
investigations are fact-finding proceedings with no
formal issues and no adverse parties … and are not
conducted for the purpose of determining the rights
or liabilities of any person.”
U.S. Code § 1154

12.
Similar Organizations
● Italy –Agenzia nazionale per la
Sicurezza del Volo (ANSV)
● Canada – Transportation Safety Board
of Canada (TSB)
● Indonesia- Komite Nasional
Keselamatan Transportasi (NTSC)
● Netherlands – Dutch Safety Board
(DSB)
● Australia – Australian Transport Safety
Bureau (ATSB)
● United Kingdom – Air Accidents
Investigation Branch (AAIB)
● Germany – Bundesstelle für
Flugunfalluntersuchung
● France –Bureau d’Enquetes et
d’Analyses pour la Securite de
l’Aviation Civile (BEA)

13.
NTSB Investigation Process

14.
NTSB Investigation Process
1. Pre-Investigation Preparation
2. Notification & Initial Response
3. On-Scene Activities
4. Post-On-Scene Activities

15.
1. Pre-Investigation
Preparation

16.
Pre-Investigation Preparation
GO TEAM
● Go team: On call investigators ready for
assignments
● Investigator-In-Change (IIC) pre-assigned
● Full Go team may contain several subject
matter experts; e.g.
○ Human performance
○ Aircraft performance
○ Air Traffic Control

17.
Pre-Investigation Preparation
GO TEAM ROSTER
● Oncall roster made available internally
○ Phone & Pager numbers
● Updated weekly
● All personnel should be able to arrive at an
airport 2 hours after notification
○ Should have essentials on them if they
live far away from an airport
● Division Chiefs responsible for testing pager

18.
2. Notification & Initial
Response

19.
Notification & Initial Response
REGIONAL RESPONSE
1. Regional office notifies headquarters of
incident
2. Closest regional office to accident will
provide at least one investigator to perform
PR & “stakedown”

20.
Notification & Initial Response
HEADQUARTERS RESPONSE
1. After incident occurs: communication center
advises IIC and chief of Major Investigations
(who subsequently inform their superiors)
2. OAS director decides whether to launch a
Go-Team
3. Other executives are made aware by Chief of
Major Investigations

21.
Notification & Initial Response
NOTIFICATION & ASSIGNMENTS
● Go-Team composition determined by
incident circumstances
● Send more specialists if in doubt

22.
Notification & Initial Response
PARTY NOTIFICATION
● IIC gives party status to organizations that
can provide technical assistance (airlines,
aircraft manufacturers etc.)
● Communication center will help with travel
arrangements and on-site administrative
support
● Go-Team will travel together to accident site

23.
3. On-Scene Activities

24.
On-Scene Activities
COMMAND ROOMS
● Have meeting rooms to accommodate at least
30 people
● Have space for media
● Ensure you have equipment in command
room
○ PCs
○ Telephone systems
○ Forms
● IIC is responsible for managing this

25.
On-Scene Activities
COMMAND ROOMS
● For Major investigations, Administrative
support is provided
● Government purchase card is available for
goods or services

26.
On-Scene Activities
ORGANIZATIONAL MEETING
● Share preliminary information
● Organize (assign) participants
● Organize observers
● Establish lines of authority

27.
“The manner in which the IIC conducts the
organizational meeting will establish the tone of the
investigation. Therefore, the importance of being
organized, articulate, assertive, composed, and
understanding cannot be overstated”
Major Investigations Manual Sec 3.2

28.
On-Scene Activities
ACCIDENT SITE SAFETY PRECAUTIONS
● Safety officer identifies & classifies risks and
then develops counter-measures
● Safety officer performs daily briefings to
accident site team.

29.
On-Scene Activities
OBSERVERS
● Observers may be allowed if they do not have
self-interest
● May include:
○ Congressional oversight committee(s)
○ Military personnel
○ Foreign Governments
○ Federal Agencies

30.
On-Scene Activities
LINE OF AUTHORITY
● IIC is the most senior person on-scene and all
investigative activity is under his/ her control
● If IIC cannot resolve an issue, IIC may talk to
Chief of Major Investigations
● Ability to escalate further if required

31.
On-Scene Activities
PROGRESS MEETINGS
● On-site progress meetings are held daily to:
○ Disseminate information obtained
○ Plan the day’s activities
○ Discuss plans for subsequent
investigative activities
● Generally start at 6pm
● Plan next day’s meeting

32.
On-Scene Activities
DAILY ACTIVITIES OF IIC
● Headquarters briefing
● Safety board staff meeting
● Party coordinator meeting
● Site visit

33.
4. Post-On-Scene Activities

34.
NTSB Report Structure
Gathering facts
about the incident
Factual
Information
Extra information
Appendices
Analyze how the
facts contribution to
the incident
Analysis
Draw conclusions
about what
happened
Conclusions
Write detailed
recommendations
Recommendations

35.
Post-On-Scene Activities
WORK PLANNING
● Discuss activities that will follow the on-scene
phase of investigation
● Build timelines for work
● Provides avenues for various teams to work
together

36.
Post-On-Scene Activities
FACTS & ANALYSIS REPORT
● A factual report based on the field notes and
subsequent investigation activities
● Each group chairman shall submit an analysis
report based on the information contained in
his or her factual report.

37.
Post-On-Scene Activities
PUBLIC HEARING
● Led by IIC/ Hearing Officer
● Identify witnesses whose testimony is
appropriate
● The witnesses may be from the parties to the
investigation or can be suggested by one or
more of the parties.
● Purpose: To ensure all relevant information is
gathered before writing the report

38.
Post-On-Scene Activities
TECHNICAL REVIEW
● Provides an additional opportunity for all
parties to review all factual information
● Ensures all issues are resolved
● Technical Review is held as soon as possible
after public hearing

39.
Post-On-Scene Activities
PREPARATION OF FINAL REPORT
● Dedicated department to help write report
● Follows a standard template
○ Annex 13 to the International Civil
Aviation Organization (ICAO)
● Contains formal recommendations to
manufacturers/ transportation authorities

40.
Recommendations &
Most Wanted List

41.
Recommendations & Most Wanted List
● NTSB advocates for particular action items
based on report(s):
○ Generally directed towards Transport
bodies/ manufacturers
● NTSB publicly tracks response of the
responsible body
https://www.ntsb.gov/safety/mwl/Pages/default.aspx

42.
How this relates to all of us?

43.
1. Pre-Investigation
Preparation

44.
Applying this to operations
PRE-INCIDENT PREPARATION
● Have an Incident commander pre-assigned
● Publish on-call schedules
○ Manager is responsible
● Test on-call pagers regularly
● Ensure that you can respond within SLA
● Printed copy of Oncall contact info
● DR
http://i.imgur.com/wvg8IDq.gif

45.
2. Notification & Initial
Response

46.
Applying this to operations
NOTIFICATION & INITIAL RESPONSE
● NOC/ SiteOps teams notifies incident
commander + manager
○ Prod-SRE gets engaged
● Prod-SRE Manager/Oncall
○ Access, Engage, Notify, Mitigate
https://docs.microsoft.com/en-us/windows/uwp/design/shell/tiles-and-notifications/images/toast-mirroring.gif

47.
Applying this to operations
NOTIFICATION & INITIAL RESPONSE
● Once verified, we launch full response for Major
Incident
● Incident commander gives “party status” to
observers
● Manager informs executives & PR
○ Periodic updates
● Mitigate
http://www.roadrunneremaillogin.com/wp-content/uploads/2018/06/RoadRunner-Email.jpg

48.
3. On-Scene Activities

49.
Applying this to operations
ON-SCENE ACTIVITIES
● Private + Public slack work-channels
● IC is empowered to make decisions
● Organizational call to ensure:
○ Problem is understood
○ Area of investigations assigned
http://www.gpla.com/static/img/projects/ubisofts-e3-social-media-war-room/war-room.gif

50.
Applying this to operations
ON-SCENE ACTIVITIES
● War room
○ Incident commander drives the war-
room
○ Roles & responsibilities assigned to each
“party”
○ Communication at regular cadence to
execs
○ Admin ensures supplies and food
● Gathering data and updating timeline doc
http://www.gpla.com/static/img/projects/ubisofts-e3-social-media-war-room/war-room.gif

51.
4. Post-On-Scene Activities

52.
Applying this to operations
POST ON-SCENE ACTIVITIES
● Post mortem
○ Dedicated team
○ PM Template
○ Blameless
● “Postmortem rollup”
○ Action items are prioritized
○ Weekly reporting on status of action-
items
https://www.economist.com/sites/default/files/imagecache/1280-width/20180414_OFP021.gif

53.
Recommendations:
Most Wanted List

54.
Applying this to operations
MOST WANTED LIST
● Use the post-incident process to improve
and hold people accountable for action
items
● Keep track of recurring issues/ repeaters
https://clip2art.com/images/meeting-clipart-animated-gif-2.gif

55.
Final Thoughts

56.
Final Thoughts
Complete Incident +
Postmortem process
NTSB Investigative
Process
The more you put in,
the more you’ll get
out
Invest
Accountability for
improvements/
action items
Accountability

57.
Questions?