1. Mistakes were made
Selena Deckelmann
@selenamarie
So
O mS
eCC
O on
N
fer
20
en
11
e c

2. This goes out to all the
sysadmins.
So
O mS
eCC
O on
N
fer
20
en
11
e c

3. You can never think
about failure too much.
So
O mS
eCC
O on
N
fer
20
en
11
e c

4. c e
11
en
20
fer
O on
N
eCC
O mS
So

5. Some goals around
pessimism.
So
O mS
eCC
O on
N
fer
20
en
11
e c

6. Plan for the worst.
c e
11
en
20
fer
O on
N
eCC
O mS
So

7. Minimize risk.
c e
11
en
20
fer
O on
N
eCC
O mS
So

8. Recover, gracefully.
c e
11
en
20
fer
O on
N
eCC
O mS
So

9. Plan for the worst.
Minimize risk.
Recover, gracefully.
So
O mS
eCC
O on
N
fer
20
en
11
e c

10. c e
11
en
20
fer
O on
N
eCC
O mS
So

11. c e
11
en
20
fer
O on
N
eCC
O mS
So

12. Tales of failure to...
Document
Test
Verify
Imagine
Implement
So
O mS
eCC
O on
N
fer
20
en
11
e c

13. Failure to document.
c e
11
en
20
fer
O on
N
eCC
O mS
So

14. Moving Day
Thanks, David Prior!
So
O mS
eCC
O on
N
fer
20
en
11
e c

15. Prevent documentation
failures.
• Write documentation.
• Update documentation.
• Make documenting a step in your written
process.
• Assign a fixed amount of time to that step.
So
O mS
eCC
O on
N
fer
20
en
11
e c

16. Documentation tools
• Graphic designers. (Pretty wikis. Pretty
docs. (Sphinx?) Diagrams.)
• Timelines.
• Bug tracking.
• Ordered todo lists.
So
O mS
eCC
O on
N
fer
20
en
11
e c

17. Failure to test.
c e
11
en
20
fer
O on
N
eCC
O mS
So

18. “My first day posing as a sysadmin
(~1990, no previous training....) I
deleted all zero length files on a Sun
workstation.”
So
O mS
eCC
O on
N
fer
20
en
11
e c

19. Prevent testing failures.
• Verify success criteria.
• Write tests.
• Test with a buddy.
• Have a plan.
So
O mS
eCC
O on
N
fer
20
en
11
e c

20. Testing tools
• Your favorite test framework
• Repeatable shell scripts
• Staging environments
So
O mS
eCC
O on
N
fer
20
en
11
e c

21. Failure to verify.
c e
11
en
20
fer
O on
N
eCC
O mS
So

22. “What does ‘-d’ actually do?”
So
O mS
eCC
O on
N
fer
20
en
11
e c

23. Prevent verification
failures.
• Have a plan for things going wrong.
• Have a staging environment.
• Test your rollback plan, not just your
implementation plan.
So
O mS
eCC
O on
N
fer
20
en
11
e c

24. Verification tools
• Staging environments
• Your buddy
So
O mS
eCC
O on
N
fer
20
en
11
e c

25. Failure to imagine.
c e
11
en
20
fer
O on
N
eCC
O mS
So

26. For my group the
bottom line was
"don't trust anyone".
So
Thanks, Maggie!
O mS
eCC
O on
N
fer
20
en
11
e c

27. Recover from failures
to imagine.
• Share your stories of failure.
• Talk with people who are different from
you.
• Act out implementation scenarios.
So
O mS
eCC
O on
N
fer
20
en
11
e c

28. Failure to implement.
c e
11
en
20
fer
O on
N
eCC
O mS
So

29. Re-implement.
• Learn from mistakes.
So
O mS
eCC
O on
N
fer
20
en
11
e c

30. Reflection.
(or, the Post-Mortem)
So
O mS
eCC
O on
N
fer
20
en
11
e c

31. Before
• Document the plan with numbered steps
and a timeline.
• Test the plan and the rollback plan.
• Identify a “point of no return”.
So
O mS
eCC
O on
N
fer
20
en
11
e c

32. During
• Screen sharing: UNIX screen,VNC, etc.
• Chatroom: AIM, Campfire (scrollback!)
• Voice: Campfire, Skype,VOIP, POTS call line
• Headsets!
• Designated time-keeper.
So
O mS
eCC
O on
N
fer
20
en
11
e c

33. After
• Documentation updates
• Post-mortems to identify areas of success
and areas for improvement.
• Limit improvements to 1-2 things.
So
O mS
eCC
O on
N
fer
20
en
11
e c

34. Plan for the worst.
Minimize risk.
Recover, gracefully.
So
O mS
eCC
O on
N
fer
20
en
11
e c

35. Thanks! 20
fer
11
en
c e
O on
N
eCC
O mS
So

36. Photo credits
• Flickr: sheepguardingllama
So
O mS
eCC
O on
N
fer
20
en
11
e c