O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

DevOps Days Toronto: From 6 Months Waterfall to 1 hour Code Deploys

750 visualizações

Publicada em

Slides used for https://www.devopsdays.org/events/2017-toronto/program/andreas-grabner/

In 2011 we delivered 2 major releases of our on premise enterprise software. Market, technology and customer requirements forced us to change that in order to remain competitive.

Now – in 2017 - we are deploying and providing feature releases every 2 weeks for both our on premise and SaaS-based offering. We deploy 170 SaaS production changes per day and have a DevOps pipeline that allows us to deploy a code change within 1h if necessary.

To increase quality, we built and provide a DevOps pipeline that currently executes 31000 Unit & Integration Tests per Hour as well as 60h UI Tests per Build. Our application teams are responsible end-to-end for their features and use production monitoring to validate their deployments which allows them to find 93% of bugs in production before it impacts our end users.

In this session I explain how this transformation worked from both “Top Down” as well as “Bottom Up” in our organization. A key component was the 4 people strong DevOps Team who developed and “sell” their DevOps Pipeline to the globally distributed application teams. I will give insights into how our pipeline enables application teams to design, code, test and run a new feature for our user base.

I will also talk about the “dark moments” as change is never without friction. Both internally as well as with our customers who also had to get used to more rapid changes.

Publicada em: Software
  • Seja o primeiro a comentar

DevOps Days Toronto: From 6 Months Waterfall to 1 hour Code Deploys

  1. 1. Confidential, Dynatrace, LLC From 6 Months Waterfall to 1h Code Deploys “It was a long journey!” Andreas Grabner - May 2017 @grabnerandi
  2. 2. Before we get started … How I explain DevOps (to my parents)
  3. 3. Ship the whole box! Quality Control Back to customer 24 “Features in a Box” Very late feedback  F r u s t r a t i o n !
  4. 4. 1 “Feature at a Time” Optimize Before DeployImmediate Customer Feedback User-Driven Continuous Delivery through DevOps
  5. 5. WHO was Dynatrace in ~ 2011? And what forced us to change?
  6. 6. 2major releases/year customers deploy & operate on-prem 2011 ~250 employees #1 APM Enterprise Space New (Stack/Cloud) Technologies New & Faster Competition Some Complaining Customers (Speed) Status Quo Market Challenge Compuware Acquisition ~3600 employees Rebranding / Integrations / Expansion
  7. 7. believe in the mission impossible Bernd Greifeneder CTO @ Dynatrace 80% 20% Organization & Culture Technology His Goal: 1h from Dev to Ops
  8. 8. 2major releases/year customers deploy & operate on-prem 26 feature releases/year 500 prod deployments/day self-service online sales SaaS & Managed 2011 2017 sprint releases (continuous-delivery) 1h: Code -> Prod6months major/minor release
  9. 9. Step #1: Run Enterprise “SaaS-like”
  10. 10. NOC lessons learnt: TOO SLOW!
  11. 11. Step #2: Lift & Shift Enterprise Software into AWS
  12. 12. Developer will never do that! Operator’s job
  13. 13. Anita Engleder Dynatrace DevOps Lead First Orchestration Engine to AWS
  14. 14. Lesson #1: Velocity uncovers new bottlenecks! •Going from 6 to 1 Month Cycles •Offered to: On-Premise Customers + SaaS-Deployments • Challenge: 1GB Monolithic Download • Impact: Error prone updates • Solution: Componentize, Automate Rollout/Rollback Capability, A/B Rollout Model
  15. 15. Lesson #2: Need to Increase Sprint Quality • Sprint Reviews Done on “dynaSprint“ • Daily Builds get deployed on “dynaDay“. Sprint builds to “dynaSprint“ • If you can only show it “on your dev machine“ its NOT DONE! • Deploy Sprint Builds into our internal Production Environment • We monitor Website, Support, Licensing, Community ... With Dynatrace • If we break our own back office software we ALL feel the pain right away
  16. 16. confidential Drinking a lot of our own Champagne…
  17. 17. Lesson #3: Essential End User Feedback Loop • Which Features to Optimize? Which Features to „Phase Out“? • Allows Reducing Technical and Business Debt • Allowed us to “Call Out Sales!” for requested features nobody used!
  18. 18. Lesson #4: Automated Error Analysis • Birth of “ARCHIE” our “Automated Log Archive Analyzer” integrated with JIRA
  19. 19. Lesson #5: We started to understand “The Cloud” • What Cloud Services to use for which tasks! • It SCALES but it AIN’T CHEAP if you make a mistake! 4x $$$ to IaaS
  20. 20. Step #3: Incubation on New Stack Keep Innovating on Enterprise Stack Incubate “Start Up” on New Stack
  21. 21. Redefining the DevOps Team’s Role Acting as engineers & production managers Dynatrace Managed/SaaS Orchestration Layer DynatracePipeline Visualization Deployment Timeline Log Overview using Dynatrace Log API JIRA Integrations
  22. 22. Monitoring as Pipeline & Platform Feature Dev Perf/Test Ops Biz Faster Innovation with Quality Gates Faster Acting on Feedback Unit Perf Cont. Perf New Deploy New Capability CI CD Remove/Promote Triage/Optimize Update Tests Innovate/Design $$$ Lower Costs Happy Users
  23. 23. Lesson #6: Pipeline quality + 10 min Builds https://github.com/Dynatrace/ufo
  24. 24. dynatrace.com/ ufo3d print: dynatrace.github.io/ufo
  25. 25. Dev: Shift-Left - Architectural Regression Decisions = Capturing Application Metrics + # of Images, # of JS, Load Time … + # of SQL, # of Logs, # of API Calls, # of Excepts ... == Functional Passed / Failed 31k Unit/Int-Tests / hour 60h UI-Tests / Build
  26. 26. Dev: Shift-Left - Architectural Regression Decisions Regression Baseline Every Metric of every Test Stop the Pipeline Early! https://github.com/Dynatrace/ufo
  27. 27. Perf / Test: Continuous Performance Validation “Performance Signature” for Build Nov 16 “Performance Signature” for Build Nov 17
  28. 28. Lesson #7: Deploy, Fail & Recover Fast
  29. 29. Total Number of Users per User Experience Conversion Rate Biz: User Feedback Driven Decisions
  30. 30. New Features + Day # 1 of Mkt Push Overall increase of Users! Jump in Conversion Rate! Biz: User Feedback Driven Decisions
  31. 31. Users keep growing Increase # of “tolerating” users! Lower Conversion as Day #1 Day #2 of Marketing Campaign Biz: User Feedback Driven Decisions
  32. 32. Drop in Conversion Rate Spikes in FRUSTRATED Users! Hotfix Deployment was rolled out Biz: User Feedback Driven Decisions
  33. 33. User Experience Back to Normal Jump in Conversion Rate! Fix of the Hotfix was rolled out Biz: User Feedback Driven Decisions
  34. 34. Lesson #7: Make Alerts more Actionable
  35. 35. confidential Be proud of your feature! DevOps  (No)Ops
  36. 36. Step #4: Bringing it back together Merging Development Teams Applying things “that work” on each others side!
  37. 37. confidential Dynatrace Transformation by the numbers 26 500 Feature Releases / Year Deployments / Day 31000 60h Unit & Int Tests / hour UI Tests per Build More Quality ~120 340 Code commits / day Stories per sprint More Agile 93% Production bugs found by Dev More Stability 450 99.998% Global EC2 Instances Global Availability
  38. 38. Raffle for DevOps Handbook + Echo Tweet Creative Ideas for UFO Usage to @grabnerandi http://www.dynatrace.com
  39. 39. Confidential, Dynatrace, LLC From 6 Months Waterfall to 1h Code Deploys “It was a long journey!” Andreas Grabner - May 2017 @grabnerandi THANKS

×