Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2ipH4vy. Kiran Bhattaram shares common operational antipatterns, useful tactics and stories learned in difficult situations: from a system with completely opaque fractal queues, a multitenant system with no circuit breakers, or a piece of software that requires hours of manual operations. Filmed at qconsf.com. Kiran Bhattaram is an engineer on the Observability team at Stripe. She has previously written code for the New York Times, LinkedIn, and MIT CSAIL.

1. A D A R K A N D S T O R M Y N I G H T
TA L E S O F O P E R A B I L I T Y A N T I - PA T T E R N S

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
operational-antipattern

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. Kiran Bhattaram
@kiranb

5. B U LW E R - LY T T O N
It was a dark and stormy night; the rain fell in torrents — except at
occasional intervals, when it was checked by a violent gust of wind
which swept up the streets (for it is in London that our scene lies),
rattling along the housetops, and fiercely agitating the scanty flame
of the lamps that struggled against the darkness.

6. DEFINITIONS
What is operability?
▸ The ability to keep a system in a safe and reliable
functioning condition, according to pre-defined
operational requirements.

7. Characteristics of operability
▸ safety & reliability
▸ scalability
▸ grace under pressure
DEFINITIONS
▸ ease of upgrades
▸ observability
▸ usability
▸ cultural practices around incidents
▸ AND MORE

8. DEFINITIONS
Characteristics of an operable system
▸ Converge towards a stable state.
▸ Give operators visibility and tools.
▸ Designed to be usable and unsurprising.

9. DEFINITIONS
Agenda
Robustness Usability Review!Observability

10. 1. ROBUSTNESS

11. THE TALE OF THE SYSTEM THAT
COULDN’T GIVE ANYTHING UP
STORY 1

12. ROBUSTNESS
Define your critical path.

13. ROBUSTNESS
Harvest, Yield and Scalable Tolerant Systems
Yield =
successful requests
total requests
!= uptime
Harvest =
data available
total data
* dropping requests
* degrading response

14. ROBUSTNESS
Controlling yield: load shedding upstream requests
▸ categories of load shedders:
▸ # of requests
▸ # of concurrent requests (protect against the long tail)
▸ overall fleet utilization (keep x% of workers for core
traffic)

15. ROBUSTNESS
Controlling harvest: circuit breakers
▸ stop calling a dependency if it seems down!
▸ what do you return?
▸ cached data
▸ nil
▸ or propagate the error upstream

16. ROBUSTNESS
Controlling harvest: circuit breakers & compartmentalization
http://idighardware.com/2013/10/fire-doors-everything-you-always-wanted-to-know-but-were-afraid-to-ask/

17. ROBUSTNESS
Putting it all together: giving things up
▸ Combine harvest/yield degradation in different ways to
protect the critical path
▸ Monitor any degradation!
▸ Dark launch your rate limiters to check what they’d block.

18. ROBUSTNESS
Robustness, in review
▸ know how the system sheds
load
▸ know how it reacts to
downstream failures
Converge to a stable state.

19. 2. OBSERVABILITY

20. THE TALE OF THE
FRACTAL QUEUE
STORY 2

21. OBSERVABILITY
Instrument EVERYTHING
▸ especially with queues
▸ percentiles, not averages
▸ don’t intermingle logs (keep a searchable trace ID on
requests)

22. OBSERVABILITY
Over-collect data, but build dashboards carefully
▸ work metrics
▸ is the system doing the thing it’s supposed to?
▸ resource metrics
▸ how are the components of the system behaving?
▸ build your dashboard with work metrics first.

23. THE TALE OF THE 64
ALERT WEEK
STORY 4

24. OBSERVABILITY
Don’t normalize deviance

25. OBSERVABILITY
Knowing what to alert on
▸ Monitor the alert volume of your system!
▸ Pages should be actionable and represent user pain.

26. OBSERVABILITY
Observability: what we learned
▸ Kiran has a special vendetta against unmonitored queues.
▸ Building good dashboards: work metrics & resource
metrics.
▸ Monitor alert volume, too!

27. 3. USABILITY

28. 6. Recognition vs. recall
9. Help users recognize,
diagnose, and recover from
errors
USABILITY
A quick side note: Nielsen Heuristics
1. Visibility of system status
2. Match between system and the
real world
3. User control and freedom
4. Consistency and standards
5. Error prevention
6. Recognition vs. recall
7. Flexibility and efficiency of use
8. Aesthetic and minimalist design
9. Help users recognize, diagnose,
and recover from errors
10. Help and documentation
1. Visibility of system status
3. User control and freedom
5. Error prevention

29. Story 5: the tale of the special snowflake service

30. USABILITY
Heuristic 4. Consistency and Standards
▸ pattern-matching across
similar systems is really
valuable!
▸ Choose boring
technology: spend your
innovation tokens wisely!

31. OBSERVABILITY
Heuristic 3. User control and freedom
▸ Tooling is a part of the service!
▸ relatedly, deploy mechanisms are related to availability!
▸ Give operators the ability to change operational
parameters.

32. THE TALE OF THE OPS
SPELL BOOK
STORY 6

33. USABILITY
Heuristic 6. Recognition v. recall
▸ Keep checklists minimal and heavily automated.
▸ long flowcharts in a runbook are :(
▸ relatedly: scripting user communications is helpful.

34. USABILITY
Heuristic 1. Visibility of system status
▸ which of these are changes to production?
▸ config changes
▸ deploys
▸ utility script runs
▸ failovers
▸ adding/decreasing capacity

35. THE TALE OF THE
AMBIGUOUS ERROR
MESSAGE
STORY 7

36. USABILITY
Heuristic 9. Help users recognize, diagnose, and recover from errors
▸ error messages are a crucial part of your interface
▸ Writing a good alert message:
▸ expressed in plain language, precisely indicate the
problem, and constructively suggest a solution (runbooks!)
▸ (ex.) CRITICAL: Served 5% 5xx results in the last 5 minutes!
<link to runbook>

37. USABILITY
Usability, in review
▸ Operational experience matters! Consider:
▸ whether the system follows general conventions.
▸ how it alerts operators to errors clearly and
unambiguously.
▸ how minimal and usable the tooling is.

38. Review
▸ Robustness
▸ Does your system converge to a stable state?
▸ Observability
▸ Can you infer what the internal state of the system looks like?
▸ Usability
▸ Do your operators have control over the state of the system?
Do you adhere to general standards?
REVIEW

39. THE TALE OF THE SAD
QUEUE
STORY THE LAST
:(

40. A DARK AND STORMY
NIGHT
STORY THE LAST

41. Resources
▸ Harvest, Yield, and Scalable Tolerant Systems (Brewer & Fox)
▸ How Complex Systems Fail (Cook)
▸ "Going solid": a model of system dynamics and consequences for patient safety (Cook)
▸ Nielsen’s Usability Heuristics
▸ Choose Boring Technology (Dan McKinley)
▸ Site Reliability Engineering: How Google Runs Production Systems
▸ Stripe’s (upcoming) rate limiting blog post
▸ Collection of postmortems (Dan Luu)
REVIEW

42. REVIEW
On Designing and Deploying Internet-Scale Services, James Hamilton
▸ list of best practices, from design, to upgrades, to incident
response

43. T H A N K S !
Thanks to Ines Sombra, Charity Majors, Alyssa Frazee, Rachel Sanders, and
Andy Bonventre for review!

44. APPENDIX
STUFF I COULDN’T GET TO

45. OBSERVABILITY
decouple deploys from releases
▸ get a minimal version in dark-reads into production asap
▸ corollary: have good kill switches!
▸ Know what rollbacks look like

46. OBSERVABILITY
collect operational metrics in this shadow phase
▸ Gain historical knowledge of what the system’s healthy
state looks like.
▸ Tweak your alerts and SLAs.
▸ Gameday the system! Write runbooks!

47. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
operational-antipattern