my presentation at DevOps Day 2015 Italy

1. DevOps Metrics
lies, damned lies and statistics
Gaetano Mazzanti
@mgaewsj
agile42

2. why do we need metrics?

3. why do we need metrics?
key reason: to improve

4. why do we need metrics?
decisions

5. why do we need metrics?
predictability

6. beware of
cheating
averages
fallacies

7. beware
metrics can be gamed

8. metrics & statistics still require
reasoning and visual examination
beware

9. same mean, variance & correlation
(7.50 3.75 0.816)

10. use your eyes
(same mean, variance & correlation)
median = 7.58
median = 8.14
median = 7.11
median = 7.04

11. ecological fallacy
average math score
50/100 60/100
group A
70% of people in A have
better score than B!
group B
70%

12. ecological fallacy
50/100 60/100
70%
Group A Group B
# score # score
50 45 70 43
50 55 30 100
group A group B
average math score

13. exception fallacy

14. Simpson’s paradox
Global Natural Treat
Live 108 153
Die 123 120
Natural 47% live
Treat 56% live
Women Natural Treat
Live 57 32
Die 100 57
Natural 36,3% live
Treat 36,0% live
Men Natural Treat
Live 51 121
Die 23 63
Natural 69% live
Treat 45% live

15. Simpson’s paradox
Global Natural Treat
Live 108 153
Die 123 120
Natural 47% live
Treat 56% live
Women Natural Treat
Live 57 32
Die 100 57
Natural 36,3% live
Treat 36,0% live
Men Natural Treat
Live 51 121
Die 23 63
Natural 69% live
Treat 45% live

16. which metrics?

17. deployment frequency
lead time for changes
mean time to recover
change fail rate
how IT performance
was measured
!?!?!?

18. ITIL KPIs
“ITIL Key Performance Indicators (ITIL KPIs)
are used to assess if the processes
of an IT organization are running
according to expectations”
and if not…
just kidding

19. a few ITIL KPIs… example (1/2)

20. a few ITIL KPIs… example (2/2)

21. it’s easy to get lost
in a maze of (not relevant) data

23. why?
key question about your metrics

24. what do you want
to learn?
key question about your metrics

25. loops

26. improvement loops
build/
change
measurelearn
experiment
actionable
metric
hypothesis

27. which metrics
matter to customers
no yes
end-to-end
(global)
functional
(local)
typical
ideal

28. service oriented mindset
DevOps as a service provided
to deliver value to the business

29. pizza delivery
fast delivery
accuracy and quality
predictability
what matters
to customers

30. let work flow
flow is the movement and delivery
of customer value through a process

31. derive from poor flow
slow delivery
low quality
unpredictability

32. poor flow => queues

33. just 3 metrics?
Work In Progress
Lead Time
Throughput

34. Little’s Law
Items In Queue = Arrival Rate * Waiting Time
Lead Time = Work In Progress / Throughput

35. focus on lead time

36. 0
1
2
3
4
5
6
7
8
1-Feb 3-Feb 5-Feb 7-Feb 9-Feb 11-Feb 13-Feb 15-Feb 17-Feb 19-Feb 21-Feb
scatterplot
54%
71%
88%
96%
lead time
(days) average

37. scatterplot
source ActionableMetrics book
Lead

38. scatterplot (only bugs)
source ActionableMetrics book
Lead

39. frequency distribution
source ActionableMetrics book
Lead

40. Weibull distribution

41. what to aim for

42. aging
source ActionableMetrics book

43. ___
efficiency
process efficiency =
total time
active time________
20
5
= 25%
!1!!2!!3!!4!!5!!6!!7!!8!!9!10!11!12!13!14!15!16!17!18!19!20!
elaborate do validate deliver
waiting
active

44. SLAs
Service Level Agreements
agreementexpectation
a SLA is a contract
between a service provider and the user/
customer that defines the level of service
expected from the service provider
i.e. we expect an item to flow through the process and exit
in 5 days or less with an 85% probability of success

45. SLAs – some hints
do not set a SLA without analyzing Lead Time data
do not allow a SLA to be set by someone external to
your group
do not set a SLA without collaborating with
customers and/or other stakeholders
use different SLAs for different Work Item Types

46. SLA

47. slack – avoid full utilization
absorb variations
% capacity
utilization
queuesize
queue size
grows
exponentially
at high capacity0
5
10
15
20
25
0 10 20 30 40 50 60 70 80 90 100

48. your policies shape your data
your data shape your policies

55. where to start from?

56. a possible approach
understand sources of dissatisfaction
analyze demand & capacity
discover work item types
measure flow
set SLAs
setup metric based improvement experiments
(similar to STATIK…)

57. obsessions you should have
improve process continuously
remove problems/impediments asap
get help from metrics

59. Gaetano Mazzanti
@mgaewsj
gaetano.mazzanti@agile42.com
agile42