“Every line is the perfect length if you don't measure it.” - Marty Rubin So your organization has embarked upon a transformation to be more nimble and responsive by employing the latest tools and thinking in the Agile and DevOps arena. In this transformational context, how do you know that your initiatives are effective? Empirical measurements should provide insights on business value flow and delivery efficiency, allowing teams and organizations to see how they are progressing toward achieving their goals, but all too often we find ourselves mired in measurement traps that don't quite provide the right guidance in steering our efforts. Rooted in contemporary thinking and tested in practice, this talk explores the principles of good measurement, what to measure, what not to measure, and enumerates some key metrics to help guide and inform our Agile and DevOps efforts. If done right, metrics can present a true picture of performance, and any progression, digression of these metrics can drive learning and improvement.

1. What’s Measured Improves
Metrics
That
Matter
!
Raj Indugula
raj.indugula@lithespeed.com
Rob Brown
robert.david.brown@gmail.com
Ganesh Murugan
murugan.ganesh@gmail.com

2. Agenda
• Why and What of Metrics
• Principles of Measurement
• Key Drivers & Metrics
• Key Takeaways
• Q&A
Finally, Pin Your Metric!

3. What does
mean in today’s
Lean-Agile-DevOps
world?
3

4. What is Measurement?
Measures
• I have 5 apples
• # incidents Metrics
• I have 5 more
apples than
yesterday
• % of Sev1 incidents
since feature
rollout
ObservaSon that reduces uncertainty…
4

5. Are Metrics Evil?
5

6. 6

7. What Makes A Good Metric?
hUp://www.flickr.com/photos/circasassy/7858155676/
Understandable
ComparaSve
Behavior changing
7

8. Measure
movement
towards your
business goals &
outcomes
Where are you going?
8

9. KEY METRICS

10. 1. Business Success
• Starts with Measurable Goals
• Indicators for success
• Market share
• New business service enablement
10

11. 2. Customer/User Success
• Frequency of key
transacSons
• Amount of Sme spent in
the applicaSon
• User saSsfacSon survey
results
• A/B test results
• Customer Scket volume
www.glasbergen.com
11

12. 3. Operations Success
• UpSme (availability)
• Performance (response
Sme)
• Resource uSlizaSon
• Database query Smes
• Mean Sme to detecSon
• Support
12

13. 4. Development Success
• Lead Sme for changes
(from development to
deployment)
• Deployment frequency
• % Failed deployments
• Incident severity
• Outstanding defects

14. • Average Revenue
per User
• Customer Lifetime
Value
• Daily/Monthly
Active Users
• Average Session
Time
• % Change in
Customer Volume
• Feature Metrics
• Development
Lead Time
• Idle Time
• Cycle Time
• Work in
Progress-
Technical Debt
• Rework
• Idle Time
• MTTD
• Defects
• Deployment
Lead Time
• MTTR
• % of Failed
Deployments
• Deployment
frequency,
duration
• MTTR
• Performance
- Response
Time
• Availability
• Security Pass
Rate
• Customer Ticket
Volume
• Net Promoter
Score
• Net Value Score
Deploy
Business
Success
QA
Development
Release
Customer
Success
Operate
• Release
Frequency
• Time/Cost
per Release
• Predictability
Accelerate Value Delivery
Balance speed, risk, quality & costReduce time to obtain/respond to customer
Now, All Together
14
OperaSons Success Development Success

15. Monitoring Framework
• Aggregate metrics from
all success contexts
• See the enSre system
• Enables visualizaSon,
anomaly detecSon,
trending, alerSng…
Source Adaptation: Turnbull, The Art of Monitoring, Kindle edition, chap. 2.
Business
Success
Customer
Success
Development
Success
Operations
Success
15
Event Aggregator

16. FROM THE TRENCHES

17. 16 11 19 20 3 8 4 19 6 2
88
94
99
109
118 121
128
134
138 138 138 138
16
27
46
66 69
77
81
100
106 108
0
20
40
60
80
100
120
140
160
78 (6/8-6/21) 79 (6/22-7/5) 80 (7/6-7/19) 81 (7/20-8/2) 82 (8/3-8/16) 83 (8/17-8/30) 84 (8/31-9/13) 85 (9/14-9/27) 86 (9/28-10/11) 87
(10/12-10/25)
88 (10/26-11/8)89 (11/9-11/22)
#ofStories!
Sprints !
Stories Completed Per Sprint Scope (Total Stories Planned) Total Completed (Cumulative)
Projected (Optimistic) Projected (Pessimistic) Projected (Median)
Empirical Data Drives Enquiry and Adjustment
17
Q: When are all planned stories estimated to be completed?
A: End of Sprint 90 (12/06/16)
Q: What is the total number of stories estimated to be
completed by 11/08/2016?
A: 119 Stories based on average velocity
Q: Can we deliver all current planned stories by 10/16/2016?
A: Not Likely, given the current trend
Fixed Scope
Fixed Date
Fixed Scope
& Date

18. 90
79
69
63
52
48
44
36
35
21
15 15
4
11 10
6
11
4 4
8
14
6
00
10
20
30
40
50
60
70
80
90
100
Product 1
Pending
Resolved
229
171
162 160 154
144
124
99
71
52
30 27
18
58
9 6 10
28
19 22
30
50
100
150
200
250
Product 2
Pending
Resolved
“You Build it, You Own It”
Improving Responsiveness to Issues

19. Improving Deployment Success Rate, Not Just Frequency
19
Build Daily Success Rate
Date
Build name
9/12/2016 9/13/2016 9/14/2016 9/15/2016 9/16/2016
Test Suite 1
100% 100% 100% 100% 100%
Test Suite 2
100% 100% 100% 100% 100%
Test Suite 3
100% 100% 100% 100% 100%
Test Suite 4
100% 100% 100% 100% 100%
Test Suite 5
100% 100% 100% 85% 100%
Test Suite 6
100% 100% 94% 80% 100%

20. 2%
7%
2%
6%
52% 41%
39%
26%
3%
4%
3%
3%
3%
3%
33%
28%
36%
44%
5%
20% 9%
18%
2%
7%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
% Sprint Work Breakdown
Time to Fix
Technical Debt
Production
Incidents
Product
Enhancements
Major Interface
New Capability
Other
Arch. improvement
Insight into Team Capacity and Work Items
20
Balancing New Feature Work with Maintenance and Improvement

21. KEY TAKEAWAYS

22. • Keep inventory of metrics small
• Measure outcomes, not individuals
• Monitor balanced set of metrics
• Monitor trends, not data points
• Share Widely – InteracSon encourages exploraSon
“If you measure me in an illogical way…don’t
complain about illogical behavior” - Goldratt
22

23. Next Steps
Clear Goal Well formed
metrics
Measure &
Improve
“Metrics illuminate, not indicate”
– George Dinwiddie

24. Pin Your Metric
Useful Useless

25. REFERENCES

26. • hUp://onstartups.com/tabid/3339/bid/96738/Measuring-What-MaUers-
How-To-Pick-A-Good-Metric.aspx
• hUp://www.slideshare.net/GoAtlassian/understanding-metrics-what-to-
measure-and-why-john-custy
• hUp://www.slideshare.net/TeamQualityPro/the-good-the-bad-and-the-
metrics
• hUp://devopsenterprise.io/media/DOES_forum_metrics_102015.pdf
• hUp://devops.com/2014/11/10/devops-scorecard/
• hUp://www.daScal.com/blog/9-metrics-devops-teams-tracking/
• hUp://devops.com/2015/01/26/metrics-devops/
• hUps://blog.appdynamics.com/devops/quanSfied-devops/
• hUp://www.slideshare.net/jedi4ever/devops-metrics
• hUp://www.slideshare.net/ITRevoluSon/does15-troy-magennis-and-
julia-wester-metrics-and-modeling-helping-teams-see-how-to-improve