O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

DevOps Pipelines and Metrics Driven Feedback Loops

2.499 visualizações

Publicada em

The goal behind devops is Faster Lead Times
What this really means for Software Delivery -> my Kodak/Smart Phone Analogy
How and Which Metrics to use along the Delivery Pipeline to make better decisions along the way.

Publicada em: Software
  • Seja o primeiro a comentar

DevOps Pipelines and Metrics Driven Feedback Loops

  1. 1. Metrics-Driven Pipelines or Testing & Measures in DevOps Andreas Grabner: @grabnerandi, andreas.grabner@dynatrace.com Slides: http://www.slideshare.net/grabnerandi Podcast: https://www.spreaker.com/show/pureperformance
  2. 2. @grabnerandi The Story started in 2009
  3. 3. @grabnerandi
  4. 4. @grabnerandi “The stuff we did when we were a Start Up and we All were Devs, Testers and Ops” Quote from Andreas Grabner back in 2013 @ DevOps Boston
  5. 5. @grabnerandi
  6. 6. Utmost goal: minimize cycle time (= Lead Time) timefeature cycle time minimize Users This is where you create value!
  7. 7. From the DevOps Webinar with Gene & Mark Mark Tomlinson Performance Sherpa @mark_on_task Andi Grabner Performance Advocate @grabnerandi Gene Kim, CTO Researcher and Author @RealGeneKim Webinar Recording: https://info.dynatrace.com/apm_wc_gene_kim_na_registration.html
  8. 8. High Performers Are … 200x 2,555x more frequent deployments faster lead times than their peers Source: Puppet Labs 2015 State Of DevOps Report: https://puppet.com/resources/white-paper/2016-state-of-devops-report More Agile 3x 24x lower change failure rate faster Mean Time to Recover More Reliable
  9. 9. 24 “Features in a Box” Ship the whole box! Very late feedback 
  10. 10. „1 Feature at a Time“ „Optimize before Deploy“„Immediate Customer Feedback“ Continuous Innovation and Optimization
  11. 11. DevOps Adoption
  12. 12. 700 deployments / YEAR 10 + deployments / DAY 50 – 60 deployments / DAY Every 11.6 SECONDS Innovators (aka Unicorns): Deliver value at the speed of business
  13. 13. @grabnerandi DevOps @ Target presented at Velocity, DOES and more … http://apmblog.dynatrace.com/2016/07/07/measure-frequent-successful-software-releases/ “We increased from monthly to 80 deployments per week … only 10 incidents per month … … over 96% successful! ….”
  14. 14. “We Deliver High Quality Software, Faster and Automated using New Stack“ „Shift-Left Performance to Reduce Lead Time“ Adam Auerbach, Sr. Dir DevOps https://github.com/capitalone/Hygieia & https://www.spreaker.com/user/pureperformance “… deploy some of our most critical production workloads on the AWS platform …”, Rob Alexander, CIO
  15. 15. From 0 to DevOps in 80 days Lessons learnt from shifting an on-prem to a cloud culture Bernd Greifeneder, CTO http://dynatrace.com/trial Webinar: http://ow.ly/cEYo305kFEy Podcast: http://bit.ly/pureperf
  16. 16. 2 major releases/year customers deploy & operate on-prem 26 major releases/year 170 prod deployments/day self-service online sales SaaS & Managed 2011 2016
  17. 17. 18 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015 believe in the mission impossible 6months major/minor release + intermediate fix-packs + weeks to months rollout delay sprint releases (continuous-delivery) 1h : code to production
  18. 18. @grabnerandi https://dynatrace.github.io/ufo/ “In Your Face” Data!
  19. 19. @grabnerandi Availability dropped to 0% #1: Availability -> Brand Impact
  20. 20. @grabnerandi New Deployment + Mkt Push Increase # of unhappy users! Decline in Conversion Rate Overall increase of Users! #2: User Experience -> Conversion Spikes in FRUSTRATED Users!
  21. 21. @grabnerandi #3: Resource Cons -> Cost per Feature 4x $$$ to IaaS
  22. 22. @grabnerandi #4: Performance -> Behavior
  23. 23. Dynatrace Transformation by the numbers 23x 170 more releases Deployments / Day 31000 60hUnit+Int Tests / hour UI Tests per Build More Quality ~200 340code commits / day Stories per sprint More Agile 93% Production bugs found by Dev @grabnerandi More Stability 450 99.998%Global EC2 Instances Global Availability
  24. 24. @grabnerandi Not every Sprint ends without bruises!
  25. 25. @grabnerandi Richard Dominguez Developer in Operations Prep Sportswear „In 2013 business demanded to go from monthly to daily deployments“ „80% failed!“
  26. 26. Understanding Code Complexity • 4 Millions Lines of Monolith Code • Partially coded and commented in Russian From Monolith to Microservice • Initial devs no longer with company • What to extract withouth breaking it? Shift Left Quality & Performance • No automated testing in the pipeline • Bad builds just made it into production Cross Application Impacts • Shared Infrastructure between Apps • No consolidated monitoring strategy
  27. 27. @grabnerandi Scaling an Online Sports Club Search Service 2015201420xx Response Time 2016+ 1) 2-Man Project 2) Limited Success 3) Start Expansion 4) Performance Slows Growth Users 5) Potential Decline?
  28. 28. @grabnerandi Early 2015: Monolith Under Pressure Can‘t scale vertically endlessly! May: 2.68s 94.09% CPU Bound April: 0.52s
  29. 29. @grabnerandi From Monolith to Services in a Hybrid-Cloud Front End in Geo-Distributed Cloud Scale Backend in Containers On Premise
  30. 30. @grabnerandi Go live – 7:00 a.m.
  31. 31. @grabnerandi Go live – 12:00 p.m.
  32. 32. What Went Wrong?
  33. 33. @grabnerandi 26.7s Load Time 5kB Payload 33! Service Calls 99kB - 3kB for each call! 171! Total SQL Count Architecture Violation Direct access to DB from frontend service Single search query end-to-end
  34. 34. @grabnerandi The fixed end-to-end use case “Re-architect” vs. “Migrate” to Service-Orientation 2.5s (vs 26.7) 5kB Payload 1! (vs 33!) Service Call 5kB (vs 99) Payload! 3! (vs 177) Total SQL Count
  35. 35. @grabnerandi
  36. 36. @grabnerandi You measure it! from Dev (to) Ops
  37. 37. @grabnerandi Build 17 testNewsAlert OK testSearch OK Build # Use Case Stat # APICalls # SQL Payload CPU 1 5 2kb 70ms 1 35 5kb 120ms Use Case Tests and Monitors Service & App Metrics Build 26 testNewsAlert OK testSearch OK Build 25 testNewsAlert OK testSearch OK 1 4 1kb 60ms 34 171 104kb 550ms Ops #ServInst Usage RT 1 0.5% 7.2s 1 63% 5.2s 1 4 1kb 60ms 2 3 10kb 150ms 1 0.6% 3.2s 6 75% 2.5s Build 35 testNewsAlert - testSearch OK - - - - 2 3 7kb 100ms - - - 4 80% 2.0s Continuous Innovation and Optimization Re-architecture into „Services“ + Performance Fixes Scenario: Monolithic App with 2 Key Features
  38. 38. Where to Start? Where to Go?
  39. 39. @grabnerandi
  40. 40. @grabnerandi „Always seek to Increase Flow“ „Understand and Respond to Outcome“ „Culture on Continual Experimentation“
  41. 41. @grabnerandi „Always seek to Increase Flow“ Testing: Ensure Success in The First Way Removing Bottlenecks Eliminating Technical Debt Enable Successful Cloud & Miroservices Migration Shift-Left Quality Reduce Code Complexity
  42. 42. @grabnerandiAND MANY MORE
  43. 43. Manual Code/Architectural Bottleneck Detection • Blog & YouTube Tutorial: • http://apmblog.dynatrace.com/2016/06/23/automatic-problem-detection-with-dynatrace/ • http://bit.ly/dttutorials • Metrics • # SQL, # of Same SQLs, # Threads, # Web Service/API Calls # Exceptions, # of Logs • # Bytes Transferred, Total Page Load, # of JavaScript/CSS/Images ...
  44. 44. Automated Code/Archiecture Bottleneck Detection
  45. 45. Remove Database Bottlenecks cite the database as the most common challenge or issue with application performance 88%
  46. 46. Manual Database Bottleneck Detection • Blog & YouTube Tutorial: • http://apmblog.dynatrace.com/2016/02/18/diagnosing-java-hotspots/ • http://bit.ly/dttutorials -> Database Diagnostics • Patterns • N+1 Query, Unprepared SQL, Slow SQL, Database Cache, Indices, Loading Too Much Data ...
  47. 47. Automated Database Bottleneck Detection
  48. 48. “To Deliver High Quality Working Software Faster“ „We have to Shift-Left Performance to Optimize Pipelines“ http://apmblog.dynatrace.com/2016/10/04/scaling-continuous-delivery-shift-left-performance-to-improve-lead-time-pipeline-flow/
  49. 49. = Functional Result (passed/failed) + Web Performance Metrics (# of Images, # of JavaScript, Page Load Time, ...) + App Performance Metrics (# of SQL, # of Logs, # of API Calls, # of Exceptions ...) Fail the build early!
  50. 50. Reduce Lead Time: Stop 80% of Performance Issues in your Integration Phase CI/CD: Test Automation (Selenium, Appium, Cucumber, Silk, ...) to detect functional and architectural (performance, scalabilty) regressions Perf: Performance Test (JMeter, LoadRunner, Neotys, Silk, ...) to detect tough performance issues
  51. 51. Shift-Left Performance results in Reduced Lead Time powered by Dynatrace Test Automation http://apmblog.dynatrace.com/2016/10/04/scaling-continuous-delivery-shift-left-performance-to-improve-lead-time-pipeline-flow/
  52. 52. @grabnerandi Fast Response to Outcome: Address Deployment Impact User Experience, Conversion Rate Costs and Efficiency Availability
  53. 53. @grabnerandi Real User Feedback: Building the RIGHT thing RIGHT! Experiment & innovate on new ideas Optimizing what is not perfect Removin g what nobody needs
  54. 54. Faster Lead Times to User Value! Results in Business Success!
  55. 55. Questions Slides: slideshare.net/grabnerandi Get Tools: bit.ly/dtpersonal Watch: bit.ly/dttutorials Follow Me: @grabnerandi Read More: blog.dynatrace.com Listen: http://bit.ly/pureperf Mail: andreas.grabner@dynatrace.com
  56. 56. Andreas Grabner Dynatrace Developer Advocate @grabnerandi http://blog.dynatrace.com

×