This is a slide deck accompanying a talk Arron Forde and Kevin Connaughton gave at DevOpsDays Galway in November 2018 on Site Reliability Engineering

1. Adopting SRE in Enterprise
Organizations
A Local Journey
Kevin Connaughton & Arron Forde

2. Kevin: DevOps/SRE @
kevin.connaughton@gmail.com
Arron: DevOps/SRE @
arron.forde@gmail.com

3. Agenda
Our Product
SRE definition
Our
Journey
Structure
Process
Toil
SLI
Monitoring/Metrics
Incidents

4. Webex Calling
Magic Happens

5. The choice for Site Reliability Engineering

6. “ The SRE team is responsible for the availability , latency,
performance, efficiency, change management, monitoring , emergency
response and capacity planning”

7. The most important feature of any
system is its reliability

8. Different Adoptions
Google Development Team 1
Development Team 3
Development Team 2
SRE
Team
Netflix Cross Functional Team 1
Cross Functional Team 3
Cross Functional Team 2
Same
High-Velocity,
High-Quality
Results

9. Cisco adoption
Development Team 1
Development Team 3
Development Team 2
SRE
Team
Rotation

10. Rotation Model Benefits
• Out of hours support
• Anomalies
• Dynamic Sizing

11. Process
Demos
Tech Talks
”Ops Call”
Postmortems

12. Ticket System
Prioritise
Reprioritise
Scrub
Groom
Repeat

13. Toil is the Enemy

15. Cap Operational work at 50%

16. Pipeline
Time
Manual
Effort
Jenkins
scripts
Risk and
errors
Auto deploy
decisions
Backup DC
syncing

17. Scripts used to apply our
software in AWS needed a
PIPELINE of its own!
Pipeline

18. Pipeline
Future plans
Dynamic nodes

21. Dickerson’s Hierarchy of Reliability

22. Monitoring
Reactive
Vs
Proactive

23. Dashboards

24. Metrics Flow
ELK Stack
Stats from the
phone
Edge Device
Stats
Call
processing
Stats
Tiered Metrics

25. ELK Stack
Metrics Sources
Media Quality
Analyser
Cloud Foundry
Python Microservice

27. Using the tiered metrics to set SLOs
“More than 10% of 100 largest customers are experiencing greater than 0.5% packet loss”

28. Incident Reponse Maturity
“Aware” that incidents are normal
Having well documented processes and procedures with learning
inputs to the process

29. Runbooks
Problem:
We realized organic growth of content was getting out of hand and Critical ops content was hard to
find.
Solution:
Create Runbooks and place under source control

30. War Games

32. Ops Bot

33. Smarter Paging 1. Compare text
2. Classify the failures
3. Invoke further action

34. Takeaways
• One size SRE does not fit all
• Keep identifying toil and automate it away!
• Reliability still number one throughout
• Patience: culture shifts take time
• Blueprint set, moving our process on to other projects

35. Further Reading...
Damon Edwards @ Rundeck
SREcon18
sreweekly.com

36. Questions?