Devops is usually viewed from a traditional perspective of a collaboration of Dev, Ops, and QA, driven by the change in Culture, People, and Process. But how do you know where you stand and where to move? As in almost any field, data and metrics give you the gauges and instruments. In this talk, we’ll talk about the key measurements for the DevOps transformation process and provide you with 3 metrics you can start measuring tomorrow.
5. Poll time!
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
How many of you are software engineers?
How many of you are optimists?
How many of you are self-confident in their work?
6. Dunning-Kruger Effect a.k.a. ”optimism”
People suffer from illusory superiority,
mistakenly assessing their cognitive ability as
greater than it is.
Wikipedia
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
“
7. Second-system effect a.k.a. “self-confidence”
The tendency of small, elegant, and
successful systems, to be succeeded by over-
engineered, bloated systems, due to inflated
expectations and overconfidence.
The Mythical Man-Month
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
“
26. Poll time!
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Who knows what velocity is?
Who knows what burndown chart is?
Who has a burndown chart?
Who looks at the burndown chart?
Who trusts the burndown chart?
Who knows what to do if it doesn't look right?
31. This is devops*
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Ops
QADev
Common goals,
tools, culture
Deep
specialization
*Unless you’re netfliX
**You’re not
35. How do metrics collaborate?
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected
36. How do metrics collaborate?
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected
37. How do metrics collaborate?
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected
38. How do metrics collaborate?
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected
39. How do metrics collaborate?
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected
40. How do metrics collaborate?
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected
41. How do metrics collaborate?
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected
52. How effective are we?
Samples are good enough for that
As long as they are representative
Need to be collected over time
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
54. Meet scrum team “avengers”
@jbaruch #QConLondon #datadrivendevops
jfrog.com/shownotes
55. anecdotes from the daily scrum stand-ups
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
We never get enough testing environments
from Ops, QA is suffering.
Scrum team “Avengers”
“
57. That doesn’t look good! But why?
0
5
10
15
20
25
Raj Jane Christina Helen Lee Baruch John Scott Nataly Ruppert Josh Patrick Michael
1
2
3
6
7
10
13 13
14
15
16
18
25
Days the environment stays checked out
Days checked out
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
59. anecdotes from the daily scrum stand-ups
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Tests are failing, because the test suite is not
stable, QA makes our builds fail without
reason.
Scrum team “Avengers”
“
60. Let’s gather some data!
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
62. anecdotes from the daily scrum stand-ups
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Those developers want to deploy new GC, its
tons of work, without any profit.
Scrum team “Avengers”
“
63. Let’s gather some data!
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
77. How departments measure
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Sales
Marketing
Support
HR
Finance
Anecdotes
from daily
standup
Engineering
78. Step into data-driven life
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Removes blame game
Builds accountability and trust
Creates common base for discussion