DOES16 EU - Damon Edwards and John Willis doing the devops thing...

1. Better, Faster, and Cheaper.
How?
Damon Edwards
John Willis

2. Co-Founder & Managing Partner
DTO Solutions, Inc.
Damon Edwards
@damonedwards
Director of Ecosystem Development
Docker, Inc.
John Willis
@botchagalupe

3. devopscafe.org

4. 30x 200x
more frequent
deployments
faster lead
times
60x 168x
the change
success rate
faster mean time to
recover (MTTR)
2x 50%
more likely to
exceed profitability,
market share &
productivity goals
higher market
capitalization growth
over 3 years*
High performers compared to their peers…
Data from 2014/2015 State of DevOps Report - https://puppetlabs.com/2015-devops-report
Recent IT Performance Data is Compelling

5. 30x 200x
more frequent
deployments
faster lead
times
60x 168x
the change
success rate
faster mean time to
recover (MTTR)
2x 50%
more likely to
exceed profitability,
market share &
productivity goals
higher market
capitalization growth
over 3 years*
High performers compared to their peers…
Data from 2014/2015 State of DevOps Report - https://puppetlabs.com/2015-devops-report
Recent IT Performance Data is Compelling
Faster
Higher
Quality
More
Effective

6. Fast
CheapGood
“Pick Two!”
Conventional Wisdom

7. Faster, Better, and Cheaper?

8. Faster, Better, and Cheaper. How?

9. Culture
Automation
Measurement
Sharing

10. Culture
Automation
Measurement
Sharing

11. Better, Faster, Cheaper. How?

12. Org structure that stays out of the way
Better, Faster, Cheaper. How?

13. Dev Release OpsPlanning
Application Knowledge
Operational Knowledge
Business Intent
Handoff
!
Handoff
!
Handoff
!
Ownership
but limited
Accountability
Accountability
but no
Ownership
Silos are enemy of throughput and stability

14. Dev Release OpsPlanning
Focus on reducing handoffs and improving flow
Service Aligned Delivery Teams
Service Aligned Delivery Teams
Service Aligned Delivery Teams
Environments
as a
Service
QA
as a
Service
Insights
as a
Service
Security
as a
Service
Environments
Team
QA
Team
Monitoring
& Metrics
Team
Security
Team

15. Org structure that stays out of the way
Build everything through a SDLC
Better, Faster, Cheaper. How?

16. Average Peformers: SDLC for applications
Code
Source
Repo
Config
CI
Server
Package
Repo
Operations
Console
Application
Deployment
Automation
Pre-Production
Environments
Production
Environment
Packages
Operations
Development
SOURCE
Tests
Application
Deployment
Automation

17. Code
Versioned Release
Source
Repo
Config
Env
Spec
Run-
book
Auto-
mation
CI
Server
Package
Repo
Operations
Console
Pre-Production
Environments
Production
Environment
Packages
Operations
Development
SOURCE
Dev Ops
*
Collaboration
Dev Ops
*
Execute
Operations
Procedures
Same People!!
Tests
High Performers: SDLC for everything

18. Org structure that stays out of the way
Build everything through a SDLC
Make the work visible
Better, Faster, Cheaper. How?

19. Service
now
Parts Unlimited - "Major Release 6"
Early 2014
Project
Initiation
ZRA
(finance)
Approve
Project
Monthly
Steering
Meeting
Portfolio
C-level
Steering Comittee
Provides
Input
Project
Charter
High-Level
• Stories
• Project Info
• Description
• Budget
• Schedule
PM
Stakeholders
(Tech and Biz)
Create Work
Breakdown
Work
Breakdown
(MS Proj)
High-Level
• Milestones
• Resource
Planning
3 months 3 monthsHold / Pause
Create
Requirements
(Project Meeting)
MS
Office
• Detailed Req for new
features
• Technology refreshes
• ERD (Infra req)
• DRD (Dev req)
• BRD (Biz req)
Share
Point
Create
Design
Tech
Req
Tech
Req
Tech
Req
Tech Leads
Architects Vendor Arch
Ops Arch
High-Level
Server
Tickets
3 months
Receive
Request for
Servers
Create
Server
Request
Spreadsheet
Server
Req
PM
Tixattach
Route for
Approval
Tix
1 week 1 week
• Budget
• Appropriate
Resources
DB
App or
Web
or
Approved Into Ops
Delivery
Queue
Delivery
Manager
"Matt"
Service
now
"Heads up"
Assign to
Delivery
Engineer
Delivery
Engineer
Clarify or
Confirm
Req with
Dev or
QA
1 - 6 weeks
Provision
Server
and
Rework
DBA
Validation
App/Web
Validation
Restore
Data
1 week
App
Team
App
Team
PM
Stakeholders
(Tech and Biz)
Dev Leads
4 weeks
ARB
Queue
Detailed
Analysis and
Requirements
Jira
"Stories"
Maybe
Track Ticket
Dependencies
Confluence
Pages
Team Leads
and PMs
Assign
Requirements
add more detail
for their teams
Architecture
Review Board
"Bill" plus
Architects
Working
Group
Ops
? (sometimes)
Devs, PM, Engr, QA
Development
Sprint
2 week c/t
Existing Dev
Environments
Acquire /
Prepare needed
data
Ops DBA
Service Data
Setup
(Mainframe)
"Jennifer"
Test Data
Configuration
Manager
Development Deploy to
Integration
Dev, QA
Integration &
Regression
Testing
focused on service
Scrum
Dev/QA
Integ03
Scrum
Dev/QA
Test
Link
Sprint
Review
Release
to Prod
Product Owners
(Using own
criteria)
Create
CAB ticket
or
Scrum Team Ops Team
(if legacy)
Push Deployment
to Stage
Stage
Email Notification
Jira
NewArch
Build
VMs
Jira
Ops
Service
Now
Legacy
QA Lead
PMs
QAs
End to end
testing in Prod
Prod Env
Prd
DB
Go-No Go
decision
meeting
Team Leads
Jira
Ops
By Cluster
"Remove
Feature Flag"
(if new arch)
16 weeks
6 weeks H/C: 6 3 weeks H/C: 8
4 weeks H/C:8 3 weeks H/C: 14
Data Setup Integration Testing
DEv Arch
Create
Change
Tickets
> 100
Service
Now
Compute
Net
Facility
Cabling
Storage
"Linda"
Ops PM
RESET
DELIVERY
DATE!
Steering
Comittee
Fix
Tickets!
"Linda"
Ops PM
Dev
Leadership
Assign Dev
Team
Ops Intake
Meeting
Dev
Leadership
1 week
Group
CIOs and
Arch Leads
QA
Steering
Design
Dev Breakdown
Dev / Test
Staging Release
Server Requirements Gathering
Server Approval and Assignment
Provisioning
Production Release
Initiation and Planning
Create Ops
Tickets
TS
PD
TS PD
Gaps in Requirements
• Licenses
• Dependencies on 3rd party apps
• Capacity planning always seem low
("robbing Peter to pay Paul")
• Don't purchase in advance even though
we know it's coming
Duplicate info across
different documents
EP
D
D
Procurement of physical servers can take months (lead
times for procurement plus facilities groups)
Too many Env. in on ticket
cases audit confusion
Piecemeal requests ("2 this
week, 3 next week")
1 queue for delivery team
with ~1,000 tickets at once
Capacity issues cause delay
Often told to stop
everything and do
something else
TS
D
M
TS
M W
W TS EP
H
No monitoring or backup
for some environments
30% of delivery teams time
spent "consulting" on
performance and dealing with
unfounded requests for more
capacity
3-5 days to fix
~10% S/R
H
D M
TS
H
Often skips CAB.
What CAB reviews is
often not what built
All manual setup. 1
person really knows
how. Low data quality.
Manual process with
lots of back and forth.
Many tickets with
mismatched
priorities
Mostly
manual
testing
Manual, per
cluster
Frequently down.
External service
updates take offline.
Lots of contention.
EP
M
D
PD
M W
TS
TS D
M TS
PD
M
M
S/R - 90%
S/R - 55%
S/R - 15%
D
S/R - 20%
S/R - 50%
Sometimes submits
server requests
directly to delivery Ad-hoc requests get
lost, maybe 2-3
week delays
TS
High Level
S/R - 75%
9+ months of planning before
implementation starts
(and information / requirements still
incorrect or incomplete!)
Dev and QA told to submit sever
request 6-8 weeks in advance
(only done 50% of time)
W5. New "white
glove"
engagement
model
3. Standard product catalog
("Environments on Demand")
2. Visualization of flow of work and
expected upcoming work
4. Shorten from
Design to
Implementation
1. Fully Automated Environment Provisioning
7. Small
Batches
8. Write end-to-
end customer
func. tests
11. Resolve
interface to
legacy
10. Test data
setup
automation
13. Dev Deploy to Prod for
legacy
14. Unify
change
management
tools
15. Tool
9. Service Verification test writing: shift left to Dev
(test early)
12. Remove Bottleneck and Environment Contention
(test more)
• Make the work visibile for all
• Build alignment and consensus across team
boundaries
• Empower teams to find and fix what is getting in the
way

20. Org structure that stays out of the way
Build everything through a SDLC
Make the work visible
Immutable Infrastructure Delivery
Better, Faster, Cheaper. How?

21. • Variety
• Determine your variety of offerings
based on operational efficiency and
market demand
• Velocity
• Maintain a steady flow through all
processes of the supply chain
• Variability
• Manage inconsistencies carefully to
reduce cost and improve quality
• Visibility
• Ensure the transparency of all
processes to enable continuous
learning and improvement
Immutable Delivery helps realize Toyota’s 4VL

22. Org structure that stays out of the way
Build everything through a SDLC
Make the work visible
Immutable Infrastructure Delivery
Microservices
Better, Faster, Cheaper. How?

23. Microservices
“Small Autonomous Services
That Work Together”
-Sam Newman
“Loosely Coupled Service
Oriented Architectures With
Bounded Context”
-Adrian Cockroft

24. Working with Monlithic Services
Slide stolen from Adrian Cockcroft
Release Plan
Developer
Developer
Developer
Developer
Developer
QA Release
Integration
Ops Replace Old
With New
Release
Monolithic service updates
Works well with a small number
of developers and a single
language like php, java or ruby

25. Working with Monlithic Services
Slide stolen from Adrian Cockcroft
Release Plan
Developer
Developer
Developer
Developer
Developer
QA Release
Integration
Ops Replace Old
With New
Release
Bugs
Bugs
Monolithic service updates
Works well with a small number
of developers and a single
language like php, java or ruby

26. Working with Microservices
Slide stolen from Adrian Cockcroft
Developer
Developer
Developer
Developer
Developer
Old Release Still
Running
Release Plan
Release Plan
Release Plan
Release Plan
Deploy
Feature to
Production
Deploy
Feature to
Production
Deploy
Feature to
Production
Deploy
Feature to
Production
Bugs
Deploy
Feature to
Production
Immutable microservice deployment
scales, is faster with large teams and
diverse platform components

27. Org structure that stays out of the way
Build everything through a SDLC
Make the work visible
Immutable Infrastructure Delivery
Microservices
Respect for People
Better, Faster, Cheaper. How?

28. High-Performers care for their people like strategic assets
• Diversity
• Empathy
• Areas of Worklife Scale (AWS)
It’s not just humane, it’s a business advantage

29. Recipe for burnout is inverse of recipe for success
Work overload: Job demands exceed human limits.
Lack of control: Inability to influence decisions that affect your job.
Insufficient rewards: Insufficient financial, institutional or social rewards.
Breakdown of community: Unsupportive workplace environment.
Absence of fairness: Lack of fairness in decision-making processes.
Value conflicts: Mismatch in organizational values and the individual’s values.
Source:
Dr. Christina Maslach’s six organizational risk factors
that predict burnout (via Dr. Nicole Forsgren)

30. Org structure that stays out of the way
Build everything through a SDLC
Make the work visible
Immutable Infrastructure Delivery
Microservices
Respect for People
Better, Faster, Cheaper. How?

31. Co-Founder & Managing Partner
DTO Solutions, Inc.
Damon Edwards
@damonedwards
Director of Ecosystem Development
Docker, Inc.
John Willis
@botchagalupe
devopscafe.org