Much of the value of DevOps comes from a (renewed) focus on measurement, sharing, and continuous feedback loops. In increasingly complex DevOps workflows and environments, and especially in larger, regulated, or more crystallized organizations, these core concepts become even more critical. This session will show how, by focusing on 'metrics that matter,' you can provide objective, transparent, and meaningful feedback on DevOps processes to all stakeholders. Learn from real-life examples how to use the data generated throughout application delivery to continuously identify, measure, and improve deployment speed, code quality, process efficiency, outsourcing value, security coverage, audit success, customer satisfaction, and business alignment.

1. Copyright © 2016 Splunk Inc.
Data-Driven DevOps:
Improve Velocity and
Quality of Software
Delivery with Metrics
and Feedback
Andi Mann
@AndiMann

2. Abstract (Hidden)
Much of the value of DevOps comes from a (renewed) focus on measurement, sharing,
and continuous feedback loops. In increasingly complex DevOps workflows and
environments, and especially in larger, regulated, or more crystallized organizations, these
core concepts become even more critical.
This session will show how, by focusing on 'metrics that matter,' you can provide objective,
transparent, and meaningful feedback on DevOps processes to all stakeholders. Learn from
real-life examples how to use the data generated throughout application delivery to
continuously identify, measure, and improve deployment speed, code quality, process
efficiency, outsourcing value, security coverage, audit success, customer satisfaction, and
business alignment.

3. Defining DevOps
INTEGRATION
COLLABORATION
COMMUNICATION
BETWEEN DEV AND OPS
METHODS FOR IMPROVING

4. CAMS – as close to prescriptive as DevOps gets

5. Shared Feedback Enables ‘The Three Ways’
Gene Kim, “DevOps Cookbook” and “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.”

6. DevOps Workflow is Becoming Complex and Opaque
6
Build
(Jenkins,
Bamboo)
Code
(Git,
MS-TFS)
Plan
(Jira,
Rally)
Test/QA
(Cucumber,
SonarQube)
Stage
(Pivotal,
AWS)
Release
(Jenkins,
Octopus)
Data Center
Device
Data
Engagement
Data
Config
(Puppet,
Ansible)
Monitor
(NewRelic,
Dynatrace)
Cloud Services Network Services
www/HTTP
Data
Social
Sentiment
Wire
Data
Application
Data
Continuous Integration (CI) / Continuous Delivery (CD)
Site Reliability Engineering
Business Impact Monitoring
API ServicesSecurity/Compliance

7. DevOps complexity raises risk of failure
● Slower Speed
● Longer MTTR
● Lower Quality
● Reduced Agility
● Poor Visibility
● Hard to Scale
● Increased Waste
● Impaired Collaboration
7
DevOps
From Hype Cycle for Application Services 2015, Gartner Group, July 2015, Betsy Burton, Philip Allega,
http://www.gartner.com/document/3096018

8. WHAT DATA
DRIVES
YOUR
DEVOPS
DECISIONS?

9. I’m working
super hard!!
That’s my
stapler.

10. 10
Yeah, but …
… what are you
achieving?
I’m gonna need you
to come in Sunday.

11. Gartner’s DevOps ‘Metrics that Matter’
Gartner Inc., Data-Driven DevOps: Use Metrics to Help Guide Your Journey, 29 May 2014 G00264319, Analyst(s): Cameron Haight | Tapati Bandopadhyay

12. IDC’s DevOps ‘Metrics that Matter’

13. More DevOps Metrics that Might Matter
Culture
e.g.
• Retention
• Satisfaction
• Callouts
Process
e.g.
• Idea-to-cash
• MTTR
• Deliver time
Quality
e.g.
• Tests passed
• Tests failed
• Best/worst
Systems
e.g.
• Throughput
• Uptime
• Build times
Activity
e.g.
• Commits
• Tests run
• Releases
Impact
e.g.
• Signups
• Checkouts
• Revenue

14. From every tool, every process, every component, on-prem or off
One Constant:
Machine Data

15. Common Data Fabric
15
API
SDKs UI
Other Tools
Escalation/
Collaboration
Visibility Across the Whole Dev Lifecycle
Plan Code Build Test/QA Stage Release Config Monitor

16. Common Data Fabric
16
API
SDKs UI
Server, Storage.
N/W
Server
Virtualization
Operating
Systems
Infrastructure
Applications
Mobile
Applications
Cloud Services
Other Tools
Ticketing/Help
Desk
Custom
Applications
Visibility Across the Whole Ops Environment
API Services

17. Data From Dev and Ops Tools
17

18. Data From Provisioning and Config
18

19. Data from Release Servers
19

20. Data from Infrastructure Systems
20

21. Data from Database Servers
21

22. Data from Customer Systems

23. Finding Your Metrics That Matter
Work from business backwards
Mine realtime machine data
Close the feedback loops
23

24. Outcomes

25. Data drives objective feedback loops
Velocity
Deliver on time
& on budget
IT is delivering on
time, on budget
IT and Business
Leaders
Impact
Deliver code for
business needs
IT is achieving
business goals
IT and Business Leaders,
Customers, Staff
Show you when you deliver. And when you don’t.
Quality
Deliver the
quality you
promised
We deliver a quality
experience for users
Dev and Ops
Organizations

26. Objective data enables continuous improvement
Defect
Information
Capacity
Planning
Quality
Standards
Enhancement
Requests
Integration
Requirements
Acceptance
Metrics
Service Levels
and KPIs
Application Development Test and Acceptance Production
BuildCodePlan Test/QA Stage Release Config Monitor
Infrastructure
Dependencies

27. Data clearly identifies ‘waste’
Plan
Develop
(UI)
Develop
(Db)
Develop
(M’ware)
Develop
(Backend)
Security
Test
Monitor
Build
(Prod)
Architect
Secure/
Comply
DeployAccept
Unit
Test
Document
Cap Plan
Train
Feedback
Integration
Test
Configure
System
Test
Launch
CAB
Develop
(APIs)
Budget
Build
(Dev)
Mgmt/
Tooling
W
W
W
W
W
W
W
W
W
16 40 52 35 96 40 48 24 --8 2 5 6 8 2 12

28. Data ensures transparency between all stakeholders
• Release when
ready, not a date!
• Best / worst
developers
• Best / worst
providers
• Impact of new
code on ops
• Impact of new
code on biz

29. Data-driven feedback accelerates velocity
Pivot & improve with
Continuous Insights
Product Managers
identify new
opportunities
Continuously delivered to market
… and Auditors are “happy”

30. Data-driven feedback improves quality
Code quality scans Static security scans
White BoxDeveloper
checks in code
Automated
Acceptance Tests
Dynamic Security
Scans
Black Box
“Chaos Monkey”
tests
Test Fail:
Return
Test Fail:
Return
X
X
Production
QA Prod Pattern
QA Pattern Library
Test Pass:
Promote
Test Pass:
Promote to
Production
Pattern
library used
for test and
QA

31. Data-driven feedback shows business Impact

32. Fast-feedback loop for actionable commercial insights
Data-driven DevOps allows rapid innovation
BUSINESS DEV/OPS CUSTOMERS
HOW IS OUR:
• Security?
• Quality?
• Stability?
• Performance?
• Compliance?
HOW IS OUR:
• Market Launch?
• Feature Usage?
• Marketing Changes?
• Prioritization?
• Customer Sat?

33. Summary

34. Data-driven DevOps decisions help you …
Improve
Application Velocity
Visibility across silos,
tools, and processes
exposes bugs and
bottlenecks so you
can remediate,
iterate, and innovate
faster.
Improve
Application Quality
Track quality across
multiple teams,
tools, systems, and
service providers, so
you can find and fix
more issues before
production
Improve
Application Impact
Real-time analytics
correlates
application delivery
with business goals,
so you can drive
better experience
and iterate faster

35. Further Reading
● Metrics that Matter, http://intellitect.com/devops-metrics/ - Mark
Michaelis, IntelliTect
● Gartner Inc., Data-Driven DevOps: Use Metrics to Help Guide Your
Journey, 29 May 2014 G00264319, Analyst(s): Cameron Haight | Tapati
Bandopadhyay
● DevOps and the Cost of Downtime: Fortune 1000, IDC
● Blogs.splunk.com
● Splunk.com/DevOps
● Pleasediscuss.com/andimann