1.
What the NTSB teaches us about
incident management & postmortems
​Jeff Weiner
​Chief Executive Officer
​Michael Kehoe
​Staff Site Reliability Engineer
​Nina Mushiana
​Sr Site Reliability Manager

2.
Agenda and Vision

3.
Today’s
agenda
1 Introductions
2 Background on the NTSB
3 NTSB: Investigative Process
4 Recommendations & Most Wanted List
5 How this applies to us?
6 Final thoughts

4.
Michael Kehoe
​$ /USR/BIN/WHOAMI
● Staff Site Reliability Engineer @ LinkedIn
● Production-SRE Team
● Funny accent = Australian + 4 years American

5.
Nina Mushiana
​$ /USR/BIN/WHOAMI
● Sr Site Reliability Engineer Manager @ LinkedIn
● Production-SRE Team & Site-Ops

6.
Production-SRE Team @ LinkedIn
​$ /USR/BIN/WHOAMI
● Disaster Recovery - Planning & Automation
● Incident Response – Process & Automation
● Visibility Engineering – Making use of
operational data
● Reliability Principles – Defining best practice &
automating it

7.
Incident Command System (ICS)
https://training.fema.gov/emiweb/is/icsresource/assets/reviewmaterials.pdf

8.
Background on the NTSB

9.
Background on the NTSB
​JURISDICTION
● Aviation
● Surface Transportation
● Marine
● Pipeline
● Assistance to other agencies/ governments

10.
“The NTSB shall investigate or have investigated and
establish the facts, circumstances, and cause or
probable cause of accidents…”
U.S. Code § 1131

11.
“… The Board shall report on the facts and
circumstances of each accident investigated…The
Board shall make each report available to the public
at reasonable cost…”
U.S. Code § 1131

12.
“The NTSB does not assign fault or blame for an
accident or incident…accident/incident
investigations are fact-finding proceedings with no
formal issues and no adverse parties … and are not
conducted for the purpose of determining the rights
or liabilities of any person.”
U.S. Code § 1154

13.
Similar Organizations
● Italy –Agenzia nazionale per la
Sicurezza del Volo (ANSV)
● Canada – Transportation Safety Board
of Canada (TSB)
● Indonesia- Komite Nasional
Keselamatan Transportasi (NTSC)
● Netherlands – Dutch Safety Board
(DSB)
● Australia – Australian Transport Safety
Bureau (ATSB)
● United Kingdom – Air Accidents
Investigation Branch (AAIB)
● Germany – Bundesstelle für
Flugunfalluntersuchung
● France –Bureau d’Enquetes et
d’Analyses pour la Securite de
l’Aviation Civile (BEA)

14.
NTSB Investigation Process

15.
NTSB Investigation Process
1. Pre-Investigation Preparation
2. Notification & Initial Response
3. On-Scene Activities
4. Post-On-Scene Activities

16.
1. Pre-Investigation
Preparation

17.
Pre-Investigation Preparation
​GO TEAM
● Go team: On call investigators ready for
assignments
● Investigator-In-Change (IIC) pre-assigned
● Full Go team may contain several subject
matter experts; e.g.
○ Human performance
○ Aircraft performance
○ Air Traffic Control

18.
Pre-Investigation Preparation
​GO TEAM ROSTER
● Oncall roster made available internally
○ Phone & Pager numbers
● Updated weekly
● All personnel should be able to arrive at an
airport 2 hours after notification
○ Should have essentials on them if they
live far away from an airport
● Division Chiefs responsible for testing pager

19.
2. Notification & Initial
Response

20.
Notification & Initial Response
​REGIONAL RESPONSE
1. Regional office notifies headquarters of
incident
2. Closest regional office to accident will
provide at least one investigator to perform
PR & “stakedown”

21.
Notification & Initial Response
​HEADQUARTERS RESPONSE
1. After incident occurs: communication center
advises IIC and chief of Major Investigations
(who subsequently inform their superiors)
2. OAS director decides whether to launch a
Go-Team
3. Other executives are made aware by Chief of
Major Investigations

22.
Notification & Initial Response
​NOTIFICATION & ASSIGNMENTS
● Go-Team composition determined by
incident circumstances
● Send more specialists if in doubt

23.
Notification & Initial Response
​PARTY NOTIFICATION
● IIC gives party status to organizations that
can provide technical assistance (airlines,
aircraft manufacturers etc.)
● Communication center will help with travel
arrangements and on-site administrative
support
● Go-Team will travel together to accident site

24.
3. On-Scene Activities

25.
On-Scene Activities
​COMMAND ROOMS
● Have meeting rooms to accommodate at least
30 people
● Have space for media
● Ensure you have equipment in command
room
○ PCs
○ Telephone systems
○ Forms
● IIC is responsible for managing this

26.
On-Scene Activities
​COMMAND ROOMS
● For Major investigations, Administrative
support is provided
● Government purchase card is available for
goods or services

27.
On-Scene Activities
​ORGANIZATIONAL MEETING
● Share preliminary information
● Organize (assign) participants
● Organize observers
● Establish lines of authority

28.
“The manner in which the IIC conducts the
organizational meeting will establish the tone of the
investigation. Therefore, the importance of being
organized, articulate, assertive, composed, and
understanding cannot be overstated”
Major Investigations Manual Sec 3.2

29.
On-Scene Activities
​ACCIDENT SITE SAFETY PRECAUTIONS
● Safety officer identifies & classifies risks and
then develops counter-measures
● Safety officer performs daily briefings to
accident site team.

30.
On-Scene Activities
​OBSERVERS
● Observers may be allowed if they do not have
self-interest
● May include:
○ Congressional oversight committee(s)
○ Military personnel
○ Foreign Governments
○ Federal Agencies

31.
On-Scene Activities
​LINE OF AUTHORITY
● IIC is the most senior person on-scene and all
investigative activity is under his/ her control
● If IIC cannot resolve an issue, IIC may talk to
Chief of Major Investigations
● Ability to escalate further if required

32.
On-Scene Activities
​PROGRESS MEETINGS
● On-site progress meetings are held daily to:
○ Disseminate information obtained
○ Plan the day’s activities
○ Discuss plans for subsequent
investigative activities
● Generally start at 6pm
● Plan next day’s meeting

33.
On-Scene Activities
​DAILY ACTIVITIES OF IIC
● Headquarters briefing
● Safety board staff meeting
● Party coordinator meeting
● Site visit

34.
4. Post-On-Scene Activities

35.
NTSB Report Structure
Gathering facts
about the incident
Factual
Information
Extra information
Appendices
Analyze how the
facts contribution to
the incident
Analysis
Draw conclusions
about what
happened
Conclusions
Write detailed
recommendations
Recommendation
s

36.
Post-On-Scene Activities
​WORK PLANNING
● Discuss activities that will follow the on-scene
phase of investigation
● Build timelines for work
● Provides avenues for various teams to work
together

37.
Post-On-Scene Activities
​FACTS & ANALYSIS REPORT
● A factual report based on the field notes and
subsequent investigation activities
● Each group chairman shall submit an analysis
report based on the information contained in
his or her factual report.

38.
Post-On-Scene Activities
​PUBLIC HEARING
● Led by IIC/ Hearing Officer
● Identify witnesses whose testimony is
appropriate
● The witnesses may be from the parties to the
investigation or can be suggested by one or
more of the parties.
● Purpose: To ensure all relevant information is
gathered before writing the report

39.
Post-On-Scene Activities
​TECHNICAL REVIEW
● Provides an additional opportunity for all
parties to review all factual information
● Ensures all issues are resolved
● Technical Review is held as soon as possible
after public hearing

40.
Post-On-Scene Activities
​PREPARATION OF FINAL REPORT
● Dedicated department to help write report
● Follows a standard template
○ Annex 13 to the International Civil
Aviation Organization (ICAO)
● Contains formal recommendations to
manufacturers/ transportation authorities

41.
Recommendations &
Most Wanted List

42.
Recommendations & Most Wanted List
● NTSB advocates for particular action items
based on report(s):
○ Generally directed towards Transport
bodies/ manufacturers
● NTSB publicly tracks response of the
responsible body
https://www.ntsb.gov/safety/mwl/Pages/default.aspx

43.
How this relates to all of us?

44.
1. Pre-Investigation
Preparation

45.
Applying this to operations
​PRE-INCIDENT PREPARATION
● Have an Incident commander pre-assigned
● Publish on-call schedules
○ Manager is responsible
● Test on-call pagers regularly
● Ensure that you can respond within SLA
● Printed copy of Oncall contact info
● DR
http://i.imgur.com/wvg8IDq.gif

46.
2. Notification & Initial
Response

47.
Applying this to operations
​NOTIFICATION & INITIAL RESPONSE
● NOC/ SiteOps teams notifies incident
commander + manager
○ Prod-SRE gets engaged
● Prod-SRE Manager/Oncall
○ Access, Engage, Notify, Mitigate
https://docs.microsoft.com/en-us/windows/uwp/design/shell/tiles-and-notifications/images/toast-mirroring.gif

48.
Applying this to operations
​NOTIFICATION & INITIAL RESPONSE
● Once verified, we launch full response for Major
Incident
● Incident commander gives “party status” to
observers
● Manager informs executives & PR
○ Periodic updates
● Mitigate
http://www.roadrunneremaillogin.com/wp-content/uploads/2018/06/RoadRunner-Email.jpg

49.
3. On-Scene Activities

50.
Applying this to operations
​ON-SCENE ACTIVITIES
● Private + Public slack work-channels
● IC is empowered to make decisions
● Organizational call to ensure:
○ Problem is understood
○ Area of investigations assigned
http://www.gpla.com/static/img/projects/ubisofts-e3-social-media-war-room/war-room.gif

51.
Applying this to operations
​ON-SCENE ACTIVITIES
● War room
○ Incident commander drives the
war-room
○ Roles & responsibilities assigned to each
“party”
○ Communication at regular cadence to
execs
○ Admin ensures supplies and food
● Gathering data and updating timeline doc
http://www.gpla.com/static/img/projects/ubisofts-e3-social-media-war-room/war-room.gif

52.
4. Post-On-Scene Activities

53.
Applying this to operations
​POST ON-SCENE ACTIVITIES
● Post mortem
○ Dedicated team
○ PM Template
○ Blameless
● “Postmortem rollup”
○ Action items are prioritized
○ Weekly reporting on status of
action-items
https://www.economist.com/sites/default/files/imagecache/1280-width/20180414_OFP021.gif

54.
Recommendations:
Most Wanted List

55.
Applying this to operations
​MOST WANTED LIST
● Use the post-incident process to improve
and hold people accountable for action
items
● Keep track of recurring issues/ repeaters
https://clip2art.com/images/meeting-clipart-animated-gif-2.gif

56.
Final Thoughts

57.
Final Thoughts
Complete Incident +
Postmortem process
NTSB Investigative
Process
The more you put in,
the more you’ll get
out
Invest
Accountability for
improvements/
action items
Accountability

58.
Questions?