Six months ago, Confluence Cloud’s reliability for enterprise tenants was at an all time low. This is the story of how we turned things around - starting with the fundamentals of measuring everything, re-defining metrics to be insightful of actual customer pain, auditing end-to-end resilience. The most challenging part however, has been building the reliability habit. Beyond just devops best practices, this means empowering teams to own product stability front and center through practices and tools.

1. Building the Reliability Habit:
Lessons from the trenches of a product development team
SWATI RAJU | Senior Engineering Manager | @sraju
BHAKTI MEHTA | Chief Architect | @bmehta

2. Confluence 100% outage in eu-west and
us-east
7th Oct 2019 | HOT-88841 | SEV-0

4. Agenda
Background of Confluence Cloud
Principles of reliability
Habits to build the “antifragile muscle”
Takeaways

5. Background of Confluence Cloud
Forked codebase
from server
Multi-tenant
environment
Breaking down
the monolith
Legacy features
not built for scale

6. Confluence 100% outage in eu-west and
us-east
7th Oct 2019 | HOT-88841 | SEV-0

7. Building a high quality productDriving Reliability
People
Process
Technology
Data
Tools

8. Agenda
Background of Confluence Cloud
Principles of reliability
Habits to build the “antifragile muscle”
Takeaways

9. Principle 1:
Move fast and break things
does not work

13. New process changes
Chopper process Progressive rolloutsLonger soak times

14. Principle 2:
Accurately measure the
customer pain

15. UptimeReliability

16. Calculating reliability

17. Mapping of Key Journeys

18. Principle 3:
More you sweat in the field, the
less you bleed in war

19. Sweat the details!

20. if (the unknown error) {
do nothing;
}
Sweat the details

21. A good intention with a bad
approach, often leads to
poor results.
- Thomas A. Edison

22. Principles of
reliability recap
1. Move fast and break things does not work
2. Accurately measure the customer pain
3. More you sweat in the field, the less you bleed in war

23. Agenda
Background of Confluence Cloud
Principles of reliability
Habits to build the “antifragile muscle”
Takeaways

24. We are what we repeatedly do.
Excellence then, is not an act,
but a habit.
- Will Durant

25. Habits
1. Measure the right thing
2. Expect the unexpected
3. Audit end to end resilience
4. Learn. Adapt. Evolve

26. Measure the right thing
Habit 1:

27. Measure the right thing
● Experience tracking using instrumentation for 4 core
experiences
○ View page
○ Create page
○ Edit page
○ Add page comment
● Dashboards to show constant progress
● Monitoring and measuring impact caused by Hots

28. Dashboards

29. Habit 2:
Expect the unexpected
Anticipate and plan for failures

30. Shift Left
Shift Left

31. Load Testing

32. Throttling and Limiting

33. Plan for graceful degradation

34. Habit 3:
Audit end to end resilience

35. Audit end to end resilience
Focus on early
detection
Force update
Frontend code
Attribute
properly
Focus on
trends

36. Habit 4:
Learn . Adapt . Evolve .

37. Learn . Adapt . Evolve .
Learn from Hot
incidents
Adapt to evolving
needs
Evolve into the nextgen
of Confluence

38. Reliability Improvements

39. Reliability Improvements

40. Reliability Improvements

41. Incidents will happen

42. Dogs of Confluence

43. Agenda
Background of Confluence Cloud
Principles of reliability
Habits to build the “antifragile muscle”
Takeaways

44. • Reliability is a continuous effort
• Build the culture for reliability
• When things fail, show empathy
Final
Takeway

45. Thank you!
SWATI RAJU | Senior Engineering Manager | @sraju
BHAKTI MEHTA | Chief Architect | @bmehta