1. Failure Happens
F***, the F***ing thing is F***king F***ed*
*Official WebOps term from Artur Bergman
Jesse Robbins
jesse@oreilly.com

2. This will be on the test:
FAILURE HAPPENS!

5. 25%
75%

6. 25%
75% Paranoid

7. 25% Pyromaniac
75% Paranoid

9. Good
Book!

10. “multiple and unexpected
interactions of failures are
inevitable”
-­‐Charles
Perrow

11. Failure Happens

12. define:
Nines (roughly)

13. define:
Nines (roughly)
99% 5256 min (3.5 days)

14. define:
Nines (roughly)
99% 5256 min (3.5 days)
99.9% 528 min ( 8.8 hours )

15. define:
Nines (roughly)
99% 5256 min (3.5 days)
99.9% 528 min ( 8.8 hours )
99.99% 53 min

16. define:
Nines (roughly)
99% 5256 min (3.5 days)
99.9% 528 min ( 8.8 hours )
99.99% 53 min
99.999% 5 min

17. define:
Nines (roughly)
99% 5256 min (3.5 days)
99.9% 528 min ( 8.8 hours )
99.99% 53 min
99.999% 5 min
99.9999% 30 Seconds

18. define:
Nines (roughly)
99% 5256 min (3.5 days)
99.9% 528 min ( 8.8 hours )
99.99% 53 min
99.999% 5 min
99.9999% 30 Seconds
99.99999% 3 Seconds

19. Internet Routing... won’t.

24. ;''-1(<"=/-)"3.1>0?-'"@'-':
!"#$$%"&'(')*)"+,-.,-/01,( +/.01210*"345467"89: #

26. #googlefail

27. YOU

28. Continuous Power...
isn’t

29. 365 Main SF

30. 365 364.96 Main SF

33. Failure happens
A single datacenter is the
problem
• Since they all fail at some point
Recovery procedures after
failure
• Power was gone ~45 minutes
• Most services took hours to come back
• Some unnamed ones more than 12 hours

34. Truck 1, Rackspace 0

36. Geography is a
Single Point of Failure

40. +2304,$5%67"#,-8$1
!"#$%#&'()(#*&+,&!"#$%&!'()* #%-#%*%,.&'(/,.#+%*&0+.1&-#%2+3&(/."4%*&(2&".&)%"*.&5678
!"#$%&''( +#,$-#$,%./-$0,1 )*

41. Taser weilding robbers
C I Hosts' Chicago facility
robbed twice!
(the other two times were
merely "break-ins where things
were stolen")

43. Providers are
baskets too.

46. Failure Happens.
Anyone promising otherwise
is either foolish or lying
(or both).

47. Go Here!
Jesse Robbins
jesse@oreilly.com