By Baruch Sadogursky Devops is usually viewed from a traditional perspective of a collaboration of Dev, Ops and QA, driven by the change in Culture, People and Process. But how do you know where you stand and were to move? As in almost any field, data and metrics give you the gauges and instruments. In this talk we’ll talk about the key measurements for the DevOps transformation process and provide you with 3 metrics you can start measuring tomorrow.

1. #DataDrivenDevOps
How to cook a precise
software in a modern world
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

2. About me
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

3. shownotes
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
www.jfrog.com/shownotes
Slides
Video (tomorrow!)
All the links!
Ratings, comments
Raffle!

4. Cobbler's
children have
no shoes
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

5. Poll time!
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
How many of you are software engineers?
How many of you are optimists?
How many of you are self-confident in their work?

6. Dunning-Kruger Effect a.k.a. ”optimism”
People suffer from illusory superiority,
mistakenly assessing their cognitive ability as
greater than it is.
Wikipedia
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
“

7. Second-system effect a.k.a. “self-confidence”
The tendency of small, elegant, and
successful systems, to be succeeded by over-
engineered, bloated systems, due to inflated
expectations and overconfidence.
The Mythical Man-Month
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
“

8. The consequences
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

9. disillusion
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Software I like
Software I
know really
well

10. overconfidence
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Every software is 80% ready 80% of the time
Every developer, ever
“

11. Hard to give up
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

12. @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

13. @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

14. How departments measure
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Sales
Marketing
Support
HR
Finance
Anecdotes
from daily
standup
Engineering

15. How departments measure
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Sales
Marketing
Support
HR
Finance
Anecdotes
from daily
standup
Engineering

16. @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

17. @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

18. @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

19. Engineering metrics in pre-devops era
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Dev
• Velocity
• Architecture
• Compliance
QA
• Incidents
• Defects
• MTTR
• Code coverage
• Test suite
stability
Ops
• SLA
• Cost env setup
• Average
customer cost
• Tools stability

20. DEVOPS!
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
?

21. Velocity to agile is like ? To devops…
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
?

22. Velocity to agile is like NPS To devops…
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
NPS

23. Also, it’s a lot like profits
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Metric Easily
understandable
Unity actionable
Profit
✓ ✓ ✗
Velocity
✓ ✓ ✗
NPS
✓ ✓ ✗

24. Also, it’s a lot like profits
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Metric Easily
understandable
Unity actionable
Profit
✓ ✓ ✗
Velocity
✓ ✓ ✗
NPS
✓ ✓ ✗

25. Also, it’s a lot like profits
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Metric Easily
understandable
Unity actionable
Profit
✓ ✓ ✗
Velocity
✓ ✓ ✗
NPS
✓ ✓ ✗

26. Poll time!
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Who knows what velocity is?
Who knows what burndown chart is?
Who has a burndown chart?
Who looks at the burndown chart?
Who trusts the burndown chart?
Who knows what to do if it doesn't look right?

27. What’s the problem?
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Velocity

28. True of all three!
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Metric Easily
understandable
Unity actionable
Profit
✓ ✓ ✗
Velocity
✓ ✓ ✗
NPS
✓ ✓ ✗

29. Too coarse!
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
NPS

30. Too coarse!
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
DevOps

31. This is devops*
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Ops
QADev
Common goals,
tools, culture
Deep
specialization
*Unless you’re netfliX
**You’re not

32. This is devops
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Ops
QADev

33. Same for metrics!
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Ops
QADev
NPSSpecialized
metrics

34. Same for metrics!
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Ops
QADev

35. How do metrics collaborate?
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected

36. How do metrics collaborate?
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected

37. How do metrics collaborate?
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected

38. How do metrics collaborate?
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected

39. How do metrics collaborate?
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected

40. How do metrics collaborate?
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected

41. How do metrics collaborate?
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected

42. Cooking
with
metrics
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

43. Metrics
categorization
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

44. Why?
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

45. ROI – is this activity is the right thing to do?
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

46. Continuous improvement
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

47. trust
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

48. sampling
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

49. Complete sample
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

50. @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

51. Representative sample
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

52. How effective are we?
Samples are good enough for that
As long as they are representative
Need to be collected over time
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

53. Predicting the future
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

54. Let’s get practical
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

55. Development affects operational costs
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Central container images repository
Developers keep pushing images
Storage prices skyrocket
Not all PoPs need all images

56. Continuous
improvement
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

57. Development affects QA
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Anecdotes about test suite stability
shared at daily startup
Special issue type “test suite
stability”

58. Return on
investment
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

59. Development influences Operations trust
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Operations want to predict costs and
be trustworthy
Will application updates generate
more load?

60. Trust and
predictability
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

61. Dos and Don’ts

62. Measuring the right thing
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

63. LoC LOL, anyone?
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

64. Avoid data overload
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

65. Use reliable data
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

66. Metrics
generate
incentives, be
aware!
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

67. Tweak as you go
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

68. Elevate!
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Raw Data
Product
Specific
Dashboards
Integrated
Dashboards

69. @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

70. @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

71. @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops

72. Step into data-driven life
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Removes blame game
Builds accountability and trust
Creates common base for discussion

73. Q&A and links
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
@jbaruch
#codemash
#datadrivendevops
www.jfrog.com/shownotes