Presentation on PCF Meetup on Metric Driven DevOps

1. Metrics-Driven DevOps
Myrvin Yap: myrvin.yap@dynatrace.com
Podcast: https://www.spreaker.com/show/pureperformance

2. confidential

3. Confidential, Dynatrace, LLC
Develop
Ship
Deploy
Run
Scale
Compute
nodejs mongo db netty cassandra redis
ansible jenkins puppet chef
docker cloudfoundry rh openshift rh atomic rocket
core os rancher kvm busybox
mesos marathon kubernetes swarm
amazon azure openstack mesosphere calico weave
eureka/hystrix
A whole new technology stack & polyglot development

4. Code Complexity From Monolith to Microservice
No automated testing in the pipeline Cross Application Impacts

5. confidential
24 “Features in a Box” Ship the whole box!
Very late feedback 

6. confidential
Continuous User Driven Innovation
1 “Feature at a Time”
Optimize Before DeployImmediate Customer Feedback

7. @grabnerandi
You measure it! from Dev (to) Ops

8. confidential
DevOps !=
Install Jenkins +
Micro-services & Cloud

9. @grabnerandi
Not every Sprint ends without bruises!

10. @grabnerandi
Richard Dominguez
Developer in Operations
Prep Sportswear
„In 2013 business demanded to go
from monthly to daily deployments“
„80% failed!“

11. @grabnerandi
Scaling an Online Sports Club Search Service
2015201420xx
Response Time
2016+
1) 2-Man Project 2) Limited Success
3) Start Expansion
4) Performance
Slows Growth Users
5) Potential Decline?

12. @grabnerandi
Early 2015: Monolith Under Pressure
Can‘t scale vertically endlessly!
May: 2.68s 94.09% CPU
Bound
April: 0.52s

13. @grabnerandi
From Monolith to Services in a Hybrid-Cloud
Front End in
Geo-Distributed
Cloud
Scale Backend
in Containers
On Premise

14. @grabnerandi
Go live – 7:00 a.m.

15. @grabnerandi
Go live – 12:00 p.m.

16. What Went Wrong?

17. @grabnerandi
26.7s Load Time
5kB Payload
33! Service Calls
99kB - 3kB for each call!
171! Total SQL Count
Architecture Violation
Direct access to DB from frontend service
Single search query end-to-end

18. Understanding Code Complexity
• Existing 10 year old code & 3rd party
• Skills: Not everyone is a perf expert or born architect
From Monolith to Microservice
• Service usage in the End-to-End Scenarios?
• Will it scale? Or is it just a new monolith?
Understand Deployment Complexity
• When moving to Cloud/Virtual: Costs, Latency …
• Old & new patterns, e.g: N+1 Query, Data
Understand Your End Users
• What they like and what they DONT like!
• Its priority list & input for other teams, e.g: testing

19. @grabnerandi

20. @grabnerandi
“To Deliver High Quality Working Software Faster“
„We have to Shift-Left Performance to Optimize Pipelines“
http://apmblog.dynatrace.com/2016/10/04/scaling-continuous-delivery-shift-left-performance-to-improve-lead-time-pipeline-flow/

21. Reduce Lead Time: Stop 80% of Performance
Issues in your Integration Phase
CI/CD: Test Automation (Selenium,
Appium, Cucumber, Silk, ...) to
detect functional and architectural
(performance, scalabilty) regressions
Perf: Performance Test (JMeter,
LoadRunner, Neotys, Silk, ...) to
detect tough performance issues

22. = Functional Result (passed/failed)
+ Web Performance Metrics (# of Images, # of JavaScript, Page Load Time, ...)
+ App Performance Metrics (# of SQL, # of Logs, # of API Calls, # of Exceptions ...)
Fail the build early!

23. @grabnerandi
The fixed end-to-end use case
“Re-architect” vs. “Migrate” to Service-Orientation
2.5s (vs 26.7)
5kB Payload
1! (vs 33!) Service Call
5kB (vs 99) Payload!
3! (vs 177)
Total SQL Count

24. confidential
Show time!

25. confidential
2-Tier Micro-Service App

26. confidential
Build, deploy and test this application through Jenkins

27. confidential
Push 2 Versions of the App through our Pipeline

28. confidential
Detect the regression

29. confidential
Fail the pipeline

30. confidential
Use Case: DevOps
Transformation @ Dynatrace

31. confidential
Challenges to master!
 Deploying the same way our customers do: Continuously!
 Not disrupting current operations and slower moving customers
 Aligning 300+ engineers across 3 different geos
 Solution: Innovation through Incubation!

32. 2 major releases/year
customers deploy & operate on-prem
26 major releases/year
170 prod deployments/day
self-service online sales
SaaS &
Managed
2011 2016

33. 33 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
believe in the mission impossible
6months
major/minor release
+ intermediate fix-packs
+ weeks to months
rollout delay
sprint releases (continuous-delivery)
1h : code to production

34. NOC lessons learnt

35. Shift-Left Quality
Quality/Performance matters in Dev/Staging as well!
Make Dev/CSA/PM dependent from Quality in trunk!
DevOps = start thinking like an Ops before Commit
Shift-Right Metrics
enable DEVs defining quality metrics
make DEVs to the primary consumers of their metrics

36. acting as
Engineers
Role of Dynatrace DevOps Team
Dynatrace Managed/SaaS
Orchestration Layer
DynatracePipeline Visualization
Deployment Timeline
Log Overview
using Dynatrace Log APIJIRA Integrations
&
Product Managers

37. confidential
https://github.com/Dynatrace/ufo
Raising Awareness of Pipeline Quality

38. confidential
Dev/Test: Bad Code
Changes
Ops/Biz: Unhappy End
Users
Performance:
Overloaded COEs

39. confidential
Dev/Test: Check-In
Better Code
Ops/Biz: Use Real-Time
End User Feedback
CI/CD/Performance:
Stop Bad Builds Faster

40. ruxit theme 2014.05.15
Be proud of your feature!
DevOps  NoOps

41. Dynatrace Transformation by the numbers
26
170
Releases / Year
Deployments / Day
31000 60h
Unit & Int Tests / hour UI Tests per Build
More Quality
~200 340
Code commits / day Stories per sprint
More Agile
93%
Production bugs found by Dev
More Stability 450 99.998%
Global EC2 Instances Global Availability

42. confidential
Final Thoughts …

43. confidential
% 20%
organization & culture technology

44. Dynatrace as a common platform for collaboration
Business
stakeholder
Operations
stakeholder
Apps/Dev
stakeholder

45. confidential
Ecosystem …

46. confidential

47. confidential
AI-Driven Digital Operation Ecosystem
DAVIS / REST

48. Thank you.