When apps fail, whose fault is it? In today’s DevOps world, every stakeholder in the app delivery chain is accountable for various aspects of performance, scalability, and availability. Mark Tomlinson, performance engineering veteran and founder of the popular PerfBytes podcast, and Andreas Grabner, Dynatrace performance advocate, share seven practices to help you expand performance and effective collaboration into your DevOps team, including: • Why DevOps means you need to check your ego at the door • What metrics each role across teams can focus on to build quality and performance • How to use, measure and report these metrics • What performance means for different stakeholders and the resources required • Examples of how increased collaboration and responsibility can improve performance

1. 1 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
7 Practices to Expand
Performance and Effective
Collaboration in DevOps
Performance Advocate
Andi Grabner
@grabnerandi
Performance Scientist
Mark Tomlinson
@mtomlins

2. 1 Setting the stage • What went wrong?
2 What DevOps actually
means: Check your ego
at the door
• Collaborate, Share Expertise, Take
Ownership, Be Accountable
3 Performance – what does
it mean to different
stakeholders?
• Business - How many users do we expect?
What's the accepted SLA?
• Dev - How fast does the page load on my
fancy Mac?
• Test - Does the page load correctly on my
tested browsers?
• Ops - What are the resource requirements?
4 Best Practices: Metrics
and a Facilitator
• Product Owner
• Business Analyst
• App Developer
• Functional Testers
• Performance
Testers
• Ops
• The role of the
Performance
Facilitator
A
G
E
N
D
A

3. Setting the Stage

4. 4 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
What led to these application failures?

6. Big & shiny
... is not always lean

7. 7 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
X

8. 8 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive

10. Last minute
... is not a production
deployment deal!

11. 11 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
Test Environment
Production Environment
8x slower
3x more SQL

12. 12 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
Test Environment Production Environment
Hibernate,
Classloading,
XML – The Key
Hotspots
Hibernate,
Classloading,
XML – The Key
Hotspots
I/O for Web
Requests doesn’t
even show up!
That’s Normal:
Having I/O for
Web Request as
main contributor

13. 13 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
Let’s get
started!
Tips to avoid
these situations

14. 14 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
Check your
ego at the
door

15. 15 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
Individuals don’t win a Game – plus add
in conflicting motivations . . .
Ops
CHANGE IS EVIL!

16. 16 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
Tip for your Ego
Micromanagement is DEATH to DevOps
It’s a sure sign that you’re doing something wrong

17. 17 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
It’s a Team Effort

18. 18 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
We ALL have to be one Big Team!
Common Goals
• Innovate vs Maintain
• Build what is “Right & Demanded”
• Reduce Lead Time
• Deliver High Quality
Key Ingredients
• Collaborate
• Share Expertise
• Take Ownership
• Be Accountable

19. 19 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
Old Paradigms:
Hunting for who to blame
Competing silos and departments
Centralized decision making
Hierarchical transference of power
DevOps Teams:
Everyone is to blame (and succeed)
Everyone is partially accountable
A flow of “decisions” - semi-automated
People and Automation working together
Building Accountability into Performance

20. 20 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
Tip for your Team
Good DevOps Teams operate with self-awareness, take
ownership and are accountable for their output

21. 21 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
People need to act as
filters and stop leaks and
bad input into the pipeline

22. 22 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
Performance vs
Performance

23. 23 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
•Status Quo: Bad Input to the Pipeline
• Biz/PO: Too Complex Click Paths, Too many Features in a single spot
• Design: Overloaded/Overcrowded Page Mockups, Too complex design that is too hard to
implement with available UI Frameworks
•Status Quo: Wrong Performance Thinking
•# of Features pushed in a release
•The “DevOps Way”: Think Outside your Regular Box
•Biz/PO: 80/20 rule -> don’t built the 80% features that only 20% of the users may need. Build
the 20% that 80% need in the best way!
• Biz/PO/Design: Optimize Mockup Click Paths with the feedback of End Users
• Design: Sync with your UI Devs and only mock what is technically possible
•“DevOps” Performance Metrics
• # of Interactions for a user to click through the use case -> Optimize the “Conversion Funnel”
• WPO Metrics to achieve “Lean, Clean, Responsive”
Business Analysts and Product Owners

24. 24 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
Example: Design Beautiful - Performance Metrics Shocking

25. 25 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
Tip for Business
Know the performance characteristics of the market
Involve Your End Users & Engineers in the Process
Think Lean, Clean & Responsive Design

26. 26 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
•Status Quo: Bad Input to the Pipeline
• Web Dev: Prototypes makes it to Production, Ignoring WPO, Not tested on other browsers
• App Dev: Basic Implementation Mistakes: Too many SQLs, Connection & Memory Leaks ...
• Architects: Usage of “not approved” frameworks, bad architectural decisions, e.g: Redundant
and Bad (Micro)Service Calls
•Status Quo: Current Performance Thinking
•Story Points: Delivery as many Features as requested by Business
•The “DevOps Way”: Think Outside your Regular Box
• Not just blindly implement based on Mockup. Pick the right frameworks and not the
sexiest/hottest as seen on Twitter/GitHub
• Enforce and Validate architectural rules, e.g: # of service calls, asnyc/sync executions
•“DevOps” Performance Metrics
• Works on X% of Most Commonly Used Browsers
• Resource Consumption per API Call, e.g: Memory Usage, Roundtrips to the DB, # of
Connections, # of Images …
Developers & Architects

27. 27 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
Example: MicroService Architecture Gone Bad!

28. 28 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
Tip for Engineers
Built-In Quality vs. Built-In Technical Debt
Test Across Browsers before Check-In
Include Architectural Reviews in Sprint Reviews

29. 29 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
•Status Quo: Bad Input to the Pipeline
• Don’t spam engineering with test reports of your testing tools they don’t trust
• Don’t sink time in repeated manual tests
•Status Quo: Current Performance Thinking
•Number of executed tests, number of new bugs reported
•The “DevOps Way”: Think Outside your Regular Box and Collaborate with Engineering
• Functional Testers: (R)evolutionize and Level-Up Testing to include Architectural, Scalability
and Performance Metrics - NOT JUST FUNCTIONAL GREEN/RED
• Performance Testers: Provide early feedback on performance considerations to the
engineering team. Pull in key real data points from production to learn what is really going on
out there and don’t just rely on SLAs and Load Predictions from business
•“DevOps” Performance Measure
• Number of Functional & Architectural Bug Free User Stories
• Number of Automated Tests in Continuous Integration
Software Testers

30. 30 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
Example: Functional Green But Architectural Red

31. 31 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
Tip for Testing
Level-Up: Add Dev Tools and Lingo to your work
Share Testing Expertise with Engineers
Automate, Automate, Automate

32. 32 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
•Status Quo: Bad Input to the Pipeline
• Bad/Default Configured Connection Pools (DB, Web Server, App Server, …)
• Non optimized DB indices for common SQL queries
•Status Quo: Current Performance Thinking
•Up Time, Keeping Resources (CPU, Memory, Disk, I/O …) within acceptable limits
•The “DevOps Way”: Think Outside your Regular Box and Collaborate with Engineering
• Provide input on how to Optimize SQLs during development
• Help putting app code into Stored Procedures where it makes sense
• Discuss caching “static” data in other layers then the DB
•“DevOps” Performance Measure
• Resource Consumption per new rolled out feature, e.g: CPU & I/O on Database, Bandwidth
Requirements
System and DB Admins

33. 33 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
•Status Quo: Bad Input to the Pipeline
• Wrong/Default/Forgotten Browser Cache Settings on Web Server
• CDN not properly configured
•Status Quo: Current Performance Thinking
•Availability of Key System Components
•The “DevOps Way”: Think Outside your Regular Box and Collaborate with Engineering
• Review and discuss app- & web-server deployment/configuration
• Provide web site usage monitoring and feedback to Biz
• Provide resource usage (Data Center, Cloud, CDN) insight to Architects
•“DevOps” Performance Measure
• Resource Consumption per rolled out feature, e.g: Content Size delivered from Data Center
vs. Cloud vs CDN
Operations

34. 34 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
Example: Deployment Verification via Metrics

35. 35 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
Tip for Ops
Provide Guidance during Development
Provide Feedback Loops from Ops to Dev to Test
Level-Up Teams with your Expertise

36. 36 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
These behaviors will get you arrested:
● Anti-collaborators - refusing to cooperate in
any way with the team
● Self-excusal, or ambiguous accountability -
point the finger at yourself first, before others
● Exclusive participation - keeping others
outside of the information or the decisions
● Deficient communications - leaving out
critical performance information
Summary: Avoid the Dysfunction Junction

37. 37 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
Summary: Build a DevOps Performance Pipeline

38. 38 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
• The biggest system failures have one thing in common: dysfunction.
• DevOps practices and discipline can help bring the team together.
• Leave your ego “at the door” and learn to collaborate inclusively.
• Learn how to share performance information along the flow.
• Re-define what it means to be “accountable” in your team.
• Step-up your testing practices - provide more information, sooner.
• Quantify the value of your efforts - calculate the worth of the work.
Summary...

39. 39 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
Thank You!
Mark Tomlinson
@mtomlins
http://perfbytes.com
Andi Grabner
@grabnerandi
http://blog.dynatrace.com

40. 40 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
Connect with us!
Participate in our Forum ::
community.dynatrace.com
Like us on Facebook ::
facebook.com/dynatrace
Follow us on LinkedIn ::
linkedin.com/company/dynatrace
Follow us on Twitter ::
twitter.com/dynatrace
Watch our Videos & Demos ::
youtube.com/dynatrace
Read our Blog ::
application-performance-blog.com