SRE - drupal day aveiro 2016

355 visualizações

Publicada em

Site Reliability Engineering enables agility and stability.
SREs use Software Engineering to automate themselves out of the Job.
My advice, if you want to implement this change in your company is to start with action items, alter your training and hiring, implement error budgets, do blameless postmortems and reduce toil.

Publicada em: Engenharia
  • Seja o primeiro a comentar

SRE - drupal day aveiro 2016

  1. 1. 1 ricardo.amaro@acquia.com Ricardo Amaro September 2016 Implementing Site Reliability Engineering
  2. 2. 2 ricardo.amaro@acquia.com Who am I? @Drupal @ricardoamaro Portugal Lisbon Drupal Community Family +8 years Drupal 90’s Linux Adopter 5 years at Acquia Site Reliability Engineer - Senior Tier2 Ops https://drupal.org/user/666176
  3. 3. 4 ricardo.amaro@acquia.com About Acquia Metrics ○ Acquia Cloud: ○ # of Instances (17,200+) ○ # of Production Sites (54,000+) (incl. some biggest sites in the world) ○ # API Calls (3,000 + per sec) ○ # of Availability Zones (20+) ○ # of World Regions (8)
  4. 4. 5 ricardo.amaro@acquia.com We will talk about A brief summary inspired on Google’s S.R.E. book ○ What is S.R.E? ○ Tenets of S.R.E. ○ Reliability & Toil ○ Error budget - keeping the Service Level Objective (SLO) ○ Development & Operations ○ Monitoring and Being On-Call ○ Postmortem culture - Learning from failure
  5. 5. 6 ricardo.amaro@acquia.com What is S.R.E.?
  6. 6. 7 ricardo.amaro@acquia.com ➔ Term crafted by Google in 2003. ➔ When Ben Treynor was hired to run “production” and ended up “applying software engineering to an operations function” Site Reliability Engineering
  7. 7. 8 ricardo.amaro@acquia.com ➔ SRE is taken seriously by major companies Site Reliability Engineering Microsoft Apple Amazon
  8. 8. 9 ricardo.amaro@acquia.com SREs are engineers that... ➔ Apply the principles of computer science and engineering to design and develop large, distributed computing systems. ➔ Write software for those systems alongside product developers. ➔ Build all additional pieces those systems need, like backups and load balancing. ➔ Reuse old solutions for new problems. Site Reliability Engineering
  9. 9. 10 ricardo.amaro@acquia.com “Reliability is the most fundamental feature of any product.” Ben Treynor, Google’s VP for 24/7 Operations Site Reliability Engineering
  10. 10. 11 ricardo.amaro@acquia.com DevOps & S.R.E. DevOps is a practice, which was coined around 2008, that encompasses automation of manual tasks, continuous integration and continuous delivery. It applies to a wide audience of companies whereas SRE might be considered a subset of DevOps that possesses additional skill sets. Source: https://en.wikipedia.org/wiki/Site_reliability_engineering
  11. 11. 12 ricardo.amaro@acquia.com Tenets of S.R.E.
  12. 12. 13 ricardo.amaro@acquia.com 1. Ensuring a Durable Focus on Engineering 2. Pursuing Maximum Change Velocity Without Violating SLOs 3. Monitoring 4. Emergency Response 5. Change Management 6. Demand Forecasting and Capacity Planning 7. Provisioning 8. Efficiency and Performance Tenets of SRE
  13. 13. 14 ricardo.amaro@acquia.com ➔ Hire only coders ➔ Have a Service Level Objective (SLO) for your service ➔ Measure and report performance against SLOs ➔ Use Error Budgets and gate launches on them ➔ Have a Common staffing pool for SRE and DEV ➔ Excess Ops work overflows to DEV team ➔ Cap SRE operational load at 50% and share 5% with the DEV team ➔ Oncall teams at least 8 people at one location or 6 on each, each product ➔ Maximum of 2 events per oncall shift ➔ Post mortem for every event ➔ Post mortems are BLAMELESS and focus on process and technology, not people How to achieve S.R.E. Treynor’s Action items
  14. 14. 15 ricardo.amaro@acquia.com Reliability & Toil
  15. 15. 16 ricardo.amaro@acquia.com The latest feature or That the product works? What is the most important Feature of a product?
  16. 16. 17 ricardo.amaro@acquia.com How about the “503” feature ? The most important thing is that the product works!
  17. 17. 18 ricardo.amaro@acquia.com The 80’s Waterfall software delivery model Operations @customer ➔ *Provisioning ➔ *Installing ➔ *Upgrading ➔ *Maintaining ➔ *Backups/Restore ➔ *Scaling Source: wikipedia
  18. 18. 19 ricardo.amaro@acquia.com Then came the web... ● Software as a Service ● Platform as a Service ● Cloud computing ● ... ➔ Operations overhead not on the customer side ➔ Features could now be delivered faster ➔ Customer feedback important for product improvements Product Development Ship Features Operations Users
  19. 19. 20 ricardo.amaro@acquia.com Opposite rewarding conflicts Objectives: ➔ Ship new features ➔ Launch new products Objectives: ➔ Reliability & Availability ➔ Customer success Dev Ops
  20. 20. 21 ricardo.amaro@acquia.com The problem: Toil "exhausting labour" ➔ Manual ➔ Repetitive ➔ Automatable ➔ Tactical (Unplanned work) ➔ No enduring value ➔ Scales linearly with service growth (not just “work I don’t like to do.”)
  21. 21. 22 ricardo.amaro@acquia.com An Old Solution to Toil Caption goes here ● Scale with bodies In the old operations model, you throw people at a reliability problem and keep pushing (sometimes for a year or more) until the problem either goes away or blows up in your face.
  22. 22. 23 ricardo.amaro@acquia.com As your business grows, workload trends to infinity (x) time ● Cap Ops Workload As your business grows, you need to reduce manual labor in order to continue delivering features. Put a 50% cap on Ops work and leave most of the SRE team time for writing code and reducing Toil. (y)customers/traffic Workload/Toil over time
  23. 23. 24 ricardo.amaro@acquia.com Google’s example ➔ Keep operational work (i.e., toil) below 50% of each SREs time ➔ More than 50% of each SREs time is spent on: ◆ engineering project work to reduce toil ◆ add service features - improving reliability, performance, utilization ➔ Improves career planning for the SRE ➔ Improves morale on the organization ➔ An SRE team can easily devolve into an Ops team if the 50% target is broken. Why less Toil is Better S.R.E. - A modern solution
  24. 24. 25 ricardo.amaro@acquia.com S.R.E. - A modern solution DEV + OPS ➔ This conflict is not inevitable ➔ The solution is: Error Budgets! ➔ Everyone agrees on an Error Budget (has we will explain next) ➔ SRE only prevents releases or Launches if the Error Budget is exceeded. Dev Ops
  25. 25. 26 ricardo.amaro@acquia.com Error budgets: keeping the SLOs
  26. 26. 27 ricardo.amaro@acquia.com Example: A 99.9% availability SLO means that the service can be 0.1% unavailable, which is the error budget. What is an Error Budget? The business or the product establishes Service Level Objectives (SLOs) for the system, based on Service Level indicators such as error rate, availability or latency... Error Budget
  27. 27. 28 ricardo.amaro@acquia.com ➔ 100% is the wrong reliability target for basically everything. ➔ Set an SLO that acknowledges the trade-off and leaves an error budget ➔ Error budget can be spent on anything: launching features, etc. ➔ Error budget allows for discussion about how phased rollouts and 1% experiments can maintain tolerable levels of errors. ➔ Goal of SRE team isn’t “zero outages” – SRE and product devs are incentive aligned to spend the error budget to get maximum feature velocity. ➔ Out of Budget? No problems. Do more testing between releases. How to obtain and use the Error Budget?
  28. 28. 29 ricardo.amaro@acquia.com ➔ This puts an incentive to developers that drives them to value stability (not just change). ➔ And gives control that drives SREs to permit change (not just stability). ➔ It forces decisions based on metrics, not politics- nor feelings, just data. Error Budget A Self-regulating mechanism
  29. 29. 30 ricardo.amaro@acquia.com Development & Operations
  30. 30. 31 ricardo.amaro@acquia.com ➔ Development and SRE teams share a single staffing pool ◆ If all is Reliable Devs are rewarded with teammates ◆ If Ops is overloaded, SREs are contracted to support code How are Development & Operations teams organized? Now tell me… Why should I hire you?
  31. 31. 32 ricardo.amaro@acquia.com ➔ SREs are developer/sys-admin hybrids ◆ They perform more Dev work as things become stable Development & Operations Systems, code… Are you able to cook also?
  32. 32. 33 ricardo.amaro@acquia.com ➔ SRE can only spend up to 50% of their time on ops work ➔ If operational load exceeds 50%, the ops work overflows to Dev ➔ Allow them to move to other projects Development & Operations
  33. 33. 34 ricardo.amaro@acquia.com Monitoring and Being On-Call
  34. 34. 35 ricardo.amaro@acquia.com ➔ An engineer can only react with urgency a few times a day before they get fatigued. ➔ Every page should be actionable. ➔ Every page response should require intelligence. ➔ Pages should be about a new problem or an event that hasn’t been seen before. Pager fatigue A serious a problem to be addressed
  35. 35. 36 ricardo.amaro@acquia.com Root Cause Analysis: The Core of Problem Solving and Corrective by Duke Okes https://www.amazon.com/Root-Cause-Analysis-Problem-Corrective/ dp/0873897641 Find and eliminate all root causes
  36. 36. 37 ricardo.amaro@acquia.com A healthy monitoring and alerting pipeline is simple and easy to reason about Monitoring Conclusion
  37. 37. 38 ricardo.amaro@acquia.com Postmortem culture Learning from failure
  38. 38. 39 ricardo.amaro@acquia.com ➔ Document written for ALL significant incidents ➔ Non-paged incidents are even more valuable - monitoring gaps ➔ Explain what happened in detail ➔ Find all root causes of the event ➔ Assign actions to correct the problem or improve how it is addressed next time What are Postmortems?
  39. 39. 40 ricardo.amaro@acquia.com ➔ Use a blame free postmortem culture, with the goal of exposing faults ◆ apply engineering to fix these faults, ◆ Try not just avoid or minimize them. Postmortems Are Blameless!
  40. 40. 41 ricardo.amaro@acquia.com Learn and teach with postmortems Source: http://www.xkcd.com/1495/
  41. 41. 42 ricardo.amaro@acquia.com SERIOUSLY: BLAMELESS! The Field Guide to Understanding Human Error by Sidney Dekker https://www.amazon.com/Field-Guide-Understanding-Human -Error/dp/0754648265
  42. 42. 43 ricardo.amaro@acquia.com Questions? ricardo.amaro@acquia.com September 2016
  43. 43. 44 ricardo.amaro@acquia.com The S.R.E. Google Book and more resources ● https://g.co/SREBook ● There is now #SRE on @hangops Slack. https://t.co/btPgSGkGNz to join.

×