1. THREE DAYS IN SEPTEMBER
“Houston, We Have a Problem.”
by Steve Feldman, @PerfForensics

2. The Agenda
I. This is a True Story...It Really Did Happen
II. Houston, We Have a Problem.
III. The Really Good Vendors Care
IV. Getting to Zero
V. The Damage Was Already Done
VI. Where We Are Today

11. Houston, We Have a Problem...

13. Our Outage Affected our
most Important Asset

16. Our Outage Was Caused
By Human Error

19. NEVER REBOOT A UNIX
MACHINE!

21. The Monitoring
“Cameras” Should
Always Be On

24. 24

25. 25

26. 26

27. Keep Everyone Informed

28. Who Wants their Users to
Report the Problem first?

30. Not All of the Data is
Believable

33. Crisis are the Best Time to
Determine the Strength of
the Team

40. Keep Your Boss Informed

42. Keep Your Users
Informed

44. Keep Your Users Updated

46. Continue to Keep Your
Users Updated

48. Getting to Zero

51. Log
Consolida0on

55. Continue to Keep Your
Users Updated

58. It is Not Just About
Restoring Service

59. It is OK to Admit Mistakes

61. Let Your Boss Take Credit

63. Your Boss Did Not Build a
Fragile System

64. Do a Post-Mortem

66. The Problem Started
Long Before

73. Where We are Today

74. Practice Really Matters

76. Practice Failure

78. Look at Your Manuals

80. Practice Routines
and Roles

82. Practice Everyday

86. NEVER REBOOT A UNIX
MACHINE!

87. Thanks for Listening
@PerfForensics