DevOps Means Business - Gene Kim, IT Revolution Press & Nicole Forsgren Velasquez, Utah State University

1. DevOps
Means Business
Presented by
Nicole Forsgren | Gene Kim
Findings From Puppet Labs State Of DevOps 2014
PuppetConf 2014
September 23, 2014
2014 State of DevOps

2. Puppet Labs Webinar 2014 State of DevOps
Agenda
● Demographics
● Key Findings
● Survey Methodology and Analysis
● IT Performance
● Organizational Culture
● Job Satisfaction
● Big Surprises
● Future Plans
● Q&A

3. Research Team
Gene
Kim
Multiple award-winning
CTO,
researcher and co-author
of The Phoenix
Project: A Novel About
IT, DevOps, and
Helping Your Business
Win. First time Google
Slides user.
Nicole Forsgren
Professor at the
Huntsman School of
Business at Utah State
University. IT impacts,
knowledge
management, and
analytics consultant
and researcher.
Stats ninja.
Jez
Humble
Principal at
Thoughtworks,
Lecturer at UC
Berkeley, co-author of
Continuous Delivery,
and accomplished
procrastinator.
Nigel
Kersten
CIO at Puppet Labs
and responsible for
IT/Operations there.
Almost fully recovered
sysadmin.
Puppet Labs Webinar 2014 State of DevOps

4. 9,200+
Respondents from
110 countries,
across all industries
Puppet Labs Webinar 2014 State of DevOps
Demographics
@botchagalupe#devops survey early observations #4 Retail, Banking,
Finance and Healthcare respondents LT 15% total (typically the risk averse
sectors)

5. 27% of
responses
from companies
with 500 to 9,999
employees
Size of Organization
Puppet Labs Webinar 2014 State of DevOps

6. Puppet Labs Webinar 2014 State of DevOps
Size of Infrastructure
51% of
respondents
said their
infrastructure
included
<500 Servers
@botchagalupe#devops survey early observations #1
Only 40% of respondents have more than 500 servers

7. Puppet Labs Webinar 2014 State of DevOps
Departments
16%
identified as
DevOps
Department

8. DevOps Grows Up
2012
2013
2014
What is devops?
DevOps adoption is accelerating.
Holy cow! DevOps works!
Puppet Labs Webinar 2014 State of DevOps

9. Why I Care:
IT Performance Matters
Puppet Labs Webinar 2014 State of DevOps

10. Cross-population Studies
Industry Press
Tradeshows
Blogosphere
Vendors
Consultants
Industry Analyst
Anecdote
Peer
Prior Experience
Recommendation
ROI Story Best Practice
Benchmark
Population Study
We Are Here
Academic Study
Intuitive Analytical

11. High-performing IT Orgs are
More Agile
30x 8,000x
more frequent
faster lead times
deployments
than their peers
Puppet Labs Webinar 2014 State of DevOps

12. High-performing IT Orgs are
More Reliable
Puppet Labs Webinar 2014 State of DevOps
2x 12x
the change
success rate
faster mean time to
recover (MTTR)

13. Firms with high-performing IT
orgs are winning
more likely to
exceed
profitability,
market share
& productivity
goals
50%
Puppet Labs Webinar 2014 State of DevOps
2x
higher market
capitalization
growth over 3
years*

14. How We Collected Responses
Puppet Labs Webinar 2014 State of DevOps
Snowball Sampling

15. Questions We Asked
Likert-type questions give us a more nuanced
measurement.
Puppet Labs Webinar 2014 State of DevOps

16. Latent Constructs
Some things can’t be measured directly, like happiness.
Puppet Labs Webinar 2014 State of DevOps
“Happiness Statistics” by Andrew Tarvin is licensed under CC BY 2.0

17. Quantifying IT Performance
Puppet Labs Webinar 2014 State of DevOps
Deploy
30X
more frequently
with
50%
fewer failures
● Deploy Frequency
● Mean Time to Recover (MTTR)
● Lead Time for Changes

18. Categorizing Companies: High,
Medium, Low IT Performance
Puppet Labs Webinar 2014 State of DevOps
Deploy Frequency
Mean Time to
Recover (MTTR)
Lead Time for
Changes
MTTR
Lead Time
Deploy Frequency
High-performing IT orgs
Medium-performing IT orgs
Low-performing IT orgs

19. Practices Correlated with IT
Performance Metrics
Throughput Metrics Stability Metrics
Puppet Labs Webinar 2014 State of DevOps
Deployment Frequency
● Continuous Delivery
● Version Control
Mean Time to Recover (MTTR)
● Version Control
● Monitoring System and
Application Health
Lead Time for Changes
● Version Control
● Automated Testing
Change Fail Rate
● Not strongly correlated with
specific practices.
● Significant differences between
groups with high, medium and
low change fail rates. High
performing IT orgs have 50%
lower change fail rates than
their peers.(mention Roy
Rapoport!)

20. Practices Correlated with IT
Performance Metrics
Throughput Metrics Stability Metrics
Puppet Labs Webinar 2014 State of DevOps
Deployment Frequency
● Continuous Delivery
● Version Control
Mean Time to Recover (MTTR)
● Version Control
● Monitoring System and
Application Health
Lead Time for Changes
● Version Control
● Automated Testing
Change Fail Rate
● Not strongly correlated with
specific practices.
● Significant differences between
groups with high, medium and
low change fail rates. High
performing IT orgs have 50%
lower change fail rates than
their peers.(mention Roy
Rapoport!)

21. Practices Correlated with IT
Performance Metrics
Throughput Metrics Stability Metrics
Puppet Labs Webinar 2014 State of DevOps
Deployment Frequency
● Continuous Delivery
● Version Control
Mean Time to Recover (MTTR)
● Version Control
● Monitoring System and
Application Health
Lead Time for Changes
● Version Control
● Automated Testing
Change Fail Rate
● Not strongly correlated with
specific practices.
● Significant differences between
groups with high, medium and
low change fail rates. High
performing IT orgs have 50%
lower change fail rates than
their peers.(mention Roy
Rapoport!)

22. Practices Correlated with IT
Performance Metrics
Throughput Metrics Stability Metrics
Puppet Labs Webinar 2014 State of DevOps
Deployment Frequency
● Continuous Delivery
● Version Control
Mean Time to Recover (MTTR)
● Version Control
● Monitoring System and
Application Health
Lead Time for Changes
● Version Control
● Automated Testing
Change Fail Rate
● Not strongly correlated with
specific practices.
● Significant differences between
groups with high, medium and
low change fail rates. High
performing IT orgs have 50%
lower change fail rates than
their peers.(mention Roy
Rapoport!)

23. Top Predictors of IT
Performance
• Peer-review of production changes (vs.
Puppet Labs Webinar 2014 State of DevOps
external change approval)
• Version control of all production artifacts
• Proactive monitoring of the production
environment
• High trust culture
• Win-win relationship between Dev, Ops and
Infosec
• High job satisfaction

24. Puppet Labs Webinar 2014 State of DevOps
Feedback Loops
Source: Elisabeth Hendrickson: Care And Feeding Of Feedback Loops

25. Puppet Labs Webinar 2014 State of DevOps
Organizational Culture
15% 52% 33%

26. Puppet Labs Webinar 2014 State of DevOps
Organizational Culture
Resources
● http://www.thisamericanlife.org/radio-archives/episode/403/nummi
● http://sloanreview.mit.edu/article/how-to-change-a-culture-lessons-from-nummi/
● Read Toyota Kata by Mike Rother

27. Puppet Labs Webinar 2014 State of DevOps
Job Satisfaction
Job satisfaction is the # 1
predictor of organizational
performance!
Top Predictors
of Job Satisfaction
★ High-trust organizational
culture
★ Climate of learning
★ Win-win relationships
between ops, dev and infosec
teams
★ Proactive monitoring and
autoscaling
★ Use of version control for all
production artifacts
★ Automated testing

28. Surprises
Gene
• Version control of the environment is more
important than version control of the code!
• There’s never been a better time to know
statistics
Gene (channeling Jez)
• Whether or not you have an integration or
stabilization phase has zero impact on IT perf
• Peer review is more effective than CAB
Puppet Labs Webinar 2014 State of DevOps

29. Can Large Orgs Be High
Performers?
Yes.
But orgs with 10,000+
employees 40% less likely
to be high performing vs.
500 employee orgs…
Puppet Labs Webinar 2014 State of DevOps

30. Can Large Orgs Adopt
These Practices?
Yes!
(Automated testing,
Continuous integration,
proactive monitoring…)
The only practice not being
adopted is Peer Review vs.
Change Approval!
Puppet Labs Webinar 2014 State of DevOps

31. Surprises
Nicole
• DevOps practices and IT performance impact
organizational performance
• Change fail rate wasn’t part of IT performance
Nicole (channeling Nigel)
• Forming new DevOps teams and giving people
DevOps titles is successful in practice
Puppet Labs Webinar 2014 State of DevOps

32. • Collect more stock tickers!
• 2014-15 research goals
• What do you want to know?
• Put your ideas and problems you want to
explore in this Google+
Community:http://bit.ly/FutureStateOfDevOps
Puppet Labs Webinar 2014 State of DevOps
Future Plans

33. bit.ly/puppetquestions
O&A

34. Oct 21-23 | DevOps Enterprise Summit
A conference for horses, by horses, featuring Macy’s Disney, GE Capital, and
more. Register at http://devopsenterprise.io/
Nov 9-14 | LISA ‘14
Systems engineering and ops professionals share real-world knowledge about
designing, building & maintaining critical systems.
https://www.usenix.org/conference/lisa14
Puppet Labs Webinar 2014 State of DevOps
Upcoming Events

35. Resources
• Download the 2014 State of DevOps Report:
http://puppetlabs.com/2014-devops-report
• Pre-order Jez’s forthcoming book Lean
Enterprise: Adopting Continuous Delivery,
DevOps, and Lean Startup at Scale:
http://amzn.com/1449368425
•Sign up for Gene’s newsletter to receive a
sample chapter of his forthcoming book DevOps
Cookbook:
http://itrevolution.com/blog