Damon Edwards, co-founder of Rundeck, presentation at Nike internal DevOps Day in Beaverton, OR on June 18, 2018. This talk looks at the forces that fundamentally undermine operations work and what needs to be addressed if enterprises are going to get the most out of their digital transformation and DevOps initiatives. See a Demo of Rundeck Enterprise : https://www.rundeck.com/see-demo --or-- Download Rundeck Open Source here: https://rundeck.com/open-source Connect: Stack Overflow community: https://stackoverflow.com/questions/tagged/rundeck Github: https://github.com/rundeck/rundeck/issues Twitter: https://twitter.com/Rundeck Facebook: https://www.facebook.com/RundeckInc/ LinkedIn: www.linkedin.com › company › rundeck-inc

1. Modern Operations:
Solving DevOps’ Last Mile Problem
Damon Edwards
@damonedwards
2018
June 18, 2016
Beaverton, OR

2. Ops Improvement
DevOps
Ops Tools
Community
Damon Edwards

3. This talk isn’t about cool technology.

4. This talk isn’t about cool technology.
Kubernetes
Istio
Lamba

5. This talk isn’t about cool technology.
Kubernetes
Istio
Lamba
Ansible
Docker
Prometheus
Chef
Collectd
AWS
Puppet
VMs
Nagios

6. This talk isn’t about cool technology.
Kubernetes
Istio
Lamba
Ansible
Docker
Prometheus
Chef
Collectd
AWS
Puppet
VMs
Nagios
???
???
???

7. This talk is about how we operate it.

8. OpsBusiness
Idea
Shorter Time-to-Market
Fast Feedback
from Users
Dev Ops
Running
Services
Improved Quality
Digital and DevOps
Availability Auditing
Security Compliance
"Go faster!"
“Open up!”
“Lock it down!”
Why now?

9. Let’s start with a true story…

10. Digital
Agile
DevOps
SRE
Cloud
Docker
Kubernetes
Microservices
CHANGE
Wow
That is cool
I wish I could
work there

11. But nobody was talking about what
happened after deployment…

12. It was just another Tuesday…

13. NOC
NOC
Biz
Manager
Escalate!
NOC NOC
NOC
(Bob)
Open
Incident
Ticket
9:30am 10:00am
NOC (Bob)
Biz Manager
Ticket
Context Wagon
Yes, but this
looks different
Hasn’t there been
some intermittent
errors this week?
v3
?!

14. NOC
(Bob)
Open
Incident
Ticket
Ticket
Biz
Manager
App-specific
SREs
“Try this.”
“Try that.”
SRE
SysAdmin
with Prod Access
(Steve)
SRE
SRE
SRE
SRE
SRE
SRE
Bridge
Call
Biz
Manager
fixed?
fixed?
NOC (Bob)
Biz Manager
NOC (Bob)
Biz Manager
SysAdmin (Steve)
7 x SRE
Ticket
Context Wagon
Ticket
Context Wagon

15. SRE
“It’s a problem
with the Foo
service”
SRE
SRE
Foo
SRE
SRE
SRE
SRE
Bridge
Call
Biz
Manager
Foo
Service
No.
NOC
(Bob)
Update
Ticket
Ticket
Foo
Lead Dev
+ add
12:00pm
NOC (Bob)
Biz Manager
Foo SRE
Ticket
Context Wagon
Can you
fix it?

16. o
Dev
Foo
Lead Dev
(Karen)
ding!
Ignore.
App
Manager
Hey did you see
that ticket?
Foo
Lead Dev
(Karen)
sigh.
I’ll take a look
I’m go
mor
pm
NOC (Bob)
Biz Manager
App Manager
Lead Dev (Karen)
Foo SRE
Scrum
Ticket
Context Wagon

17. k
Foo
Lead Dev
(Karen)
I’m going to need
more log files
Ticket
SysAdmin Team
+ add
Update
Ticket
Chat
“Can someone with
access to Foo Service
in Prod01 help me with
ticket #42516?”
SysAdmin
(Lee) Ticket
“logs
attached”
Foo
Lead Dev
(Karen)
Ticket
“no the
other ones”
Le
(K
NOC (Bob)
Biz Manager
App Manager
Lead Dev (Karen)
Foo SRE
SysAdmin (Lee)
Ticket
Context Wagon

18. Foo
Lead Dev
(Karen)
Logs
-Who restarted these services? (and why?)
-They didn’t use the correct environment
variables!
-This entire service pool needs to be restarted!
Ticket
Update
Ticket
NOC
(Bob)
Update
Ticket
Ticket
Middleware Team
+ add
“Middleware, please
urgent restart this entire
app pool with the correct
environment variable”
2:00pm
Ticket
Context W

19. ase
s entire
e correct
able”
NOC
(Bob)
Middleware
Manager
(Melissa)
No way. It’s the middle
of the day! You need
business approval.
NOC
(Bob)
Update
Ticket
Ticket
SVP for Line of
Business
+ add
(S
NOC (Bob)
Biz Manager
App Manager
Lead Dev (Karen)
Foo SRE
SysAdmin (Lee)
Middleware Manager
NOC (B
Biz Ma
App Ma
Lead D
Foo SR
Ticket
Context Wagon
Ticket
Context Wagon
2:30pm

20. Update
Ticket
Ticket
SVP for Line of
Business
+ add
SVP
(Susan)
Chief of
Staff
Tech VP
Tech VP
Update
Ticket
Ticket
“Restart approved”
Customer
impact?
Ticket
Middlewa
Manage
(Melissa
Wh
prod
5:00pm
NOC (Bob)
Biz Manager
App Manager
Lead Dev (Karen)
Foo SRE
SysAdmin (Lee)
Middleware Manager
SVP
Chief of Staff
2 x Tech VP
Ticket
Context Wagon

21. Share
point
proved”
Ticket
Middleware
Manager
(Melissa)
Who knows these
production services
the best?
Ellen!
Middleware Middleware
(Scott)
Ellen
to
Europe
office
Middleware
(Scott)
Trial and error
.doc
5:00pm
NOC (Bob)
Biz Manager
App Manager
Lead Dev (Karen)
Foo SRE
SysAdmin (Lee)
Middleware Manager
SVP
Chief of Staff
2 x Tech VP
Middleware (Scott)
Ticket
Context Wagon

22. Share
point
Middleware
(Scott)
Trial and error
.doc
NOC (Bob)
Biz Manager
App Manager
Lead Dev (Karen)
Foo SRE
SysAdmin (Lee)
Middleware Manager
SVP
Chief of Staff
2 x Tech VP
Middleware (Scott)
ket
Context Wagon
Middleware
(Scott)
Bar
Service
10 min Middleware
(Scott)
Waiting for
Acme Service
Acme startup
failed
Bar
Service
6:00pm

23. Come on.. no.no.no.
What? Why?
Middleware
(Scott)

24. Come on.. no.no.no.
What? Why?
Middleware
(Scott)

25. 8888888
Come on.. no.no.no.
What? Why?
Middleware
(Scott)

26. -Bar app startup timed out. Error says can’t
connect to Acme service.
- I looked at Acme but it seems to be running
-Is this error message correct? Why can’t Bar
connect?
Ticket
Update
Ticket
Middleware
(Scott)
Bar SRE
+ add
Bar SRE
(Linda)
Middleware
(Scott)
-URGENT: Network
connection issue
between Bar and
Acme
Ticket
Update
Ticket
Network
SRE Team
+ add
6:45
NOC (Bob)
Biz Manager
App Manager
Lead Dev (Karen)
Foo SRE
SysAdmin (Lee)
Middleware Manager
SVP
Chief of Staff
2 x Tech VP
Middleware (Scott)
Bar SRE (Linda)Ticket
Context Wagon
The new environment pre-flight
check is preventing startup.
Looks like Bar’s connection to
Acme is being blocked.

27. Bar SRE
(Linda)
Middleware
(Scott)
-URGENT: Network
connection issue
between Bar and
Acme
Ticket
Update
Ticket
Network
SRE Team
+ add
Bar
Lead Dev
6:45pm
ob)
ager
nager
ev (Karen)
E
SysAdmin (Lee)
Middleware Manager
SVP
Chief of Staff
2 x Tech VP
Middleware (Scott)
Bar SRE (Linda)
Customers are
calling. What
is going on?The new environment pre-flight
check is preventing startup.
Looks like Bar’s connection to
Acme is being blocked.
Bar
Lead Dev
(Liu)
Business
Managers
I can comment out
the test… But the
CD pipeline only
goes to QA ENV!

28. Network Dir
(Carlos)
Middleware
(Scott)
Carlos, I need a favor.
Can you escalate?Middleware
Manager
(Melissa)
Customers are
calling. What
is going on?
Last week..
Net SRE
VP
VP
Priority!
Different
Incident!
Net SRE Net SRE
Net SRE
Its the network!
Business
Managers
Your
network is
broken!
Business
Managers
We are already
working on it!
Network VPs
out
he
ly
V!

29. Network
SRE
(Hari)
The firewall is
blocking the traffic
You’ll have to take
it up with the
Firewall Team
-URGENT: Firewall is
blocking connection
between Bar and Acme
Ticket
Open
Firewall
Ticket
Firewall
Team
+ add
Firewall Engineer
(Freddie)
Middleware
(Scott)
Paging on-call…
Open bridge…
Can’t be the firewall, it hasn’t
changed since last Thursday.
No its the firewall.
8:00p
NOC (Bob)
Biz Manager
App Manager
Lead Dev (Karen)
Foo SRE
SysAdmin (Lee)
Middleware Manager
SVP
Chief of Staff
2 x Tech VP
Middleware (Scott)
Bar SRE (Linda)
Network PM (Carlos)
Network SRE (Bob)
Ticket
Context Wagon

30. Firewall Engineer
(Freddie)
Middleware
(Scott)
Firewall Engineer
(Freddie)
Middleware
(Scott)
Can’t be the firewall, it hasn’t
changed since last Thursday.
No its the firewall.
There was a rule change last
Thursday that would stop Bar
from talking to Acme.
Can you change it back?
Sure we make changes on
Thursday…
Chief of
Staff
SVP and VPs are livid… this was
supposed to be a safe change!!
Freddie, we’ve got customers calling.
ES
Em
pro
rul
Update
Firewall
Ticket
Firewall Engineer
(Freddie)
8:00pm

31. d VPs are livid… this was
sed to be a safe change!!
we’ve got customers calling.
ESCALATE:
Emergency
production firewall
rule change review
Ticket
Update
Firewall
Ticket
NetSec
+ add
Firewall Engineer
(Freddie)
Paging on-call…
NetSec
(Nicole)
This is production so I’ll have
to get others on the Network
CAB…
Chief of
Staff
Firewall
(Freddie)
Middleware
(Scott)
Customer outage!
… I’ll call SVP Susan
Middleware
Manager
VP
VP
Bar
Lead Dev
9:00pm
NOC (Bob)
Biz Manager
App Manager
Lead Dev (Karen)
Foo SRE
SysAd
Middle
SVP
Chief o
2 x Tec
Ticket
Context Wagon

32. I’ll have
Network
Chief of
Staff
Firewall
(Freddie)
Middleware
(Scott)
Customer outage!
APPROVE: Emergency
firewall rule change
Ticket
Update
Firewall
Ticket
NetSec
(Nicole)
… I’ll call SVP Susan
Middleware
Manager
VP
VP
Bar
Lead Dev
Firewall
(Freddie)
Net L2
(Bob)
Middl
(Sc
Firewall
change
Restart Bar
9:30pm
NOC (Bob)
Biz Manager
App Manager
Lead Dev (Karen)
Foo SRE
SysAdmin (Lee)
Middleware Manager
SVP
Chief of Staff
2 x Tech VP
Middleware (Scott)
Bar SRE (Linda)
Network PM (Carlos)
Network SRE (Bob)
Firewall (Freddie)
Ticket
Context Wagon
NetSec (Nicole)

33. Middleware
(Scott)
Update
Ticket
Ticket
Customer Engagement
Manager
+ add
Policy
!!
“Ready for
API tests”
9:45pm
Firewall
(Freddie)
Net L2
(Bob)
Middleware
(Scott)
Firewall
change
Restart Bar
I think we
are good!
Middleware
Manager
VP
VP
Bar
Lead Dev
You
“think?”
pm

34. et
gement
“Ready for
API tests”
Customer
Engagement
Manager
(Varsha)
NOC
(Bob)
Customer Engagement
Manager
(Varsha)
Update
Ticket
Ticket
“APIs OK”
Middleware
(Scott)
Upda
Tick
11:00pm
Ticket
Co

35. e
Ticket
“APIs OK”
Middleware
(Scott)
Update
Ticket
Ticket
“Services
restarted OK”
NOC
NOC
Lights are green…
I guess it is fixed.
Close
Ticket
NOC
(Bob)
Zzz
11:30pm
N
NOC (Bob)
Biz Manager
App Manager
Lead Dev (Karen)
Foo SRE
SysAdmin (Lee)
Middleware Manager
SVP
Chief of Staff
2 x Tech VP
Middleware (Scott)
Bar SRE (Linda)
Network PM (Carlos)
Network SRE (Bob)
Firewall (Freddie)
Ticket
Context Wagon
NetSec (Nicole)
Cust. Engmt. (Varsha)

36. e
Ticket
“APIs OK”
Middleware
(Scott)
Update
Ticket
Ticket
“Services
restarted OK”
NOC
NOC
Lights are green…
I guess it is fixed.
Close
Ticket
NOC
(Bob)
Zzz
11:30pm
N
NOC (Bob)
Biz Manager
App Manager
Lead Dev (Karen)
Foo SRE
SysAdmin (Lee)
Middleware Manager
SVP
Chief of Staff
2 x Tech VP
Middleware (Scott)
Bar SRE (Linda)
Network PM (Carlos)
Network SRE (Bob)
Firewall (Freddie)
Ticket
Context Wagon
NetSec (Nicole)
Cust. Engmt. (Varsha)
.

37. NOC
Lights are green…
I guess it is fixed.
Close
Ticket
NOC
(Bob)
Zzz
Next Day
SVP
(Susan)
Whose fault is this?!
Why are we so bad at change?
What additional processes
and approvals are you
adding to never let this
happen again?!
VP
VP
Dir
Dir
VP
Dir
VP
Scott)
da)
Carlos)
(Bob)
die)
NetSec (Nicole)
Cust. Engmt. (Varsha)

38. Later…

39. We’ve invested in Cloud, Agile,
DevOps, Containers…
Why does everything still take too
long and cost too much?
Executive Team
Our transformation has
largely ignored Ops

40. Most companies chase the symptoms…

42. Waiting Waiting
Waiting
Waiting
Waiting
Waiting
Waiting Waiting
Waiting
Waiting
Waiting
Waiting
Waiting

43. Waiting Waiting
Waiting
Waiting
Waiting
Waiting
Waiting Waiting
Waiting
Waiting
Waiting
Waiting
Defects
Defects
Defects
Defects
Defects
Waiting ! Defects

44. Manual /
Motion Manual /
Motion
Manual /
Motion
Manual /
Motion
Manual /
Motion
Waiting Waiting
Waiting
Waiting
Waiting
Waiting
Waiting Waiting
Waiting
Waiting
Waiting
Waiting
Defects
Defects
Defects
Defects
Defects
Waiting ! Defects ! Motion/Manual

45. Manual /
Motion Manual /
Motion
Manual /
Motion
Manual /
Motion
Manual /
Motion
Task
Switching
Task
Switching Task
SwitchingTask
Switching Task
Switching
Task
Switching
Task
Switching
Task
Switching
Task
Switching
Waiting Waiting
Waiting
Waiting
Waiting
Waiting
Waiting Waiting
Waiting
Waiting
Waiting
Waiting
Defects
Defects
Defects
Defects
Defects
Waiting ! Defects ! Motion/Manual ! Task Switching

46. Manual /
Motion Manual /
Motion
Manual /
Motion
Manual /
Motion
Manual /
Motion
Task
Switching
Task
Switching Task
SwitchingTask
Switching Task
Switching
Task
Switching
Task
Switching
Task
Switching
Task
Switching
Waiting Waiting
Waiting
Waiting
Waiting
Waiting
Waiting Waiting
Waiting
Waiting
Waiting
Waiting
Defects
Defects
Defects
Defects
Defects
Partially
Done
Partially
Done
Partially
Done
Partially
Done
Partially
Done
Waiting ! Defects ! Motion/Manual ! Task Switching ! Partially Done

47. Manual /
Motion Manual /
Motion
Manual /
Motion
Manual /
Motion
Manual /
Motion
Task
Switching
Task
Switching Task
SwitchingTask
Switching Task
Switching
Task
Switching
Task
Switching
Task
Switching
Task
Switching
Waiting Waiting
Waiting
Waiting
Waiting
Waiting
Waiting Waiting
Waiting
Waiting
Waiting
Waiting
Defects
Defects
Defects
Defects
Defects
Partially
Done
Partially
Done
Partially
Done
Partially
Done
Partially
Done
Extra
Process
Extra
Process
Extra
Process
Extra
Process
Extra
Process
Extra
Process
Extra
Process
Extra
Process
Waiting ! Defects ! Motion/Manual ! Task Switching ! Partially Done ! Extra Process

49. Not Here!

51. Not Here!

53. Not Here!

55. Here?

57. Even Better!

58. HC: 2 HC: 9
CW: 10
HC: 3
CW: 6
HC: 3
CW: 7
HC: 4
CW: 10
HC: 4
CW: 12
HC: 3
CW: 12
HC: 4
CW: 13
HC: 8
CW: 15
HC: 6
CW: 17
HC: 4
CW: 18
HC: 3
CW: 18
HC = Headcount
CW = in Context Wagon

59. Follow the conventional wisdom:

60. “We need better tools”
Follow the conventional wisdom:

61. “We need better tools”
“We need more people”
Follow the conventional wisdom:

62. “We need better tools”
“We need more people”
“We need more discipline and attention to detail”
Follow the conventional wisdom:

63. “We need better tools”
“We need more people”
“We need more discipline and attention to detail”
“We need more change reviews/approvals”
Follow the conventional wisdom:

64. “We need better tools”
“We need more people”
“We need more discipline and attention to detail”
“We need more change reviews/approvals”
Follow the conventional wisdom:

65. Challenge the conventional
wisdom about operations work

66. Forces That Undermine Operations
Silos Queues
Toil Low Trust

67. Forces That Undermine Operations
Silos Queues
Toil Low Trust

68. Silos
Backlog Information
PrioritiesTools

69. Backlog Information
I need X
PrioritiesTools
Silos

70. Backlog Information
I need X
PrioritiesTools
Silos
Backlog
I do X
Requests
for X
Silo A
Information
Priorities
Silo B
Tools

71. Silos cause disconnects and mismatches
Backlog Information
I need X
PrioritiesTools
Backlog
I do X
Requests
for X
Silo A
Information
Priorities
Silo B
Tools
Context
Context
Process
Process
Tooling
Tooling
Capacity
Capacity

72. 1
2
3
Silos Interfere with feedback loops

73. 1
2
3
Silos Interfere with feedback loops
Producer Consumer
Ops
Ops
Ops

74. Function A
Function B
Function C
Becomes siloed labor pools of functional specialists
Requests fulfilled by semi-
manual or manual effort
Primary management focus is
on protecting team capacity

75. Forces That Undermine Operations
Silos Queues
Toil Low Trust

76. How do we cover for our silos disconnects and mismatches?
Silo A Silo B

77. How do we cover for our silos disconnects and mismatches?
Silo A Silo B
Ticket
Queue

78. ??
Silo A Silo B
We all know how well that works
Ticket
Queue

79. Request queues are an expensive way to manage work
Ticket
Queue
Queues Create…
Longer Cycle Time
Increased Risk
More Variability
More Overhead
Lower Quality
Less Motivation
Adapted from Donald G. Reinertsen, The Principles of Product Development Flow: Second Generation Lean Product Development

80. What do queues do to value streams?

81. What do queues do to value streams?
Queue
A
Queue
B

82. What do queues do to value streams?
Queue
A
Queue
B
Queues disintegrate and
obfuscate value streams

83. Tickets queues become “snowflake makers”
??
Silo A Silo B
Ticket
Queue

84. Tickets queues become “snowflake makers”
??
Silo A Silo B
Ticket
Queue
Snowflakes
(each unique, technically acceptable but unreproducible and brittle)

85. Forces That Undermine Operations
Silos Queues
Toil Low Trust

86. Excessive toil prevents fixing the system

87. Excessive toil prevents fixing the system
“Toil is the kind of work tied to running a production
service that tends to be manual, repetitive,
automatable, tactical, devoid of enduring value, and
that scales linearly as a service grows.”
-Vivek Rau
Google

88. Excessive toil prevents fixing the system
Toil Engineering Work
E.W.Toil
Reduce toil
Improve the business ǡ
No capacity to reduce toil
No capacity to improve business
Toil at manageable percentage of capacity
Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”)

89. Excessive toil prevents fixing the system
Toil Engineering Work
E.W.Toil
Reduce toil
Improve the business ǡ
No capacity to reduce toil
No capacity to improve business
Toil at manageable percentage of capacity
Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”)

90. Toil Impacts Development As Well

91. Toil Impacts Development As Well
• 2016-2017 study of development teams
• 14 enterprises (insurance, healthcare, finance, travel, retail)
• Tech org headcount: 900 - 6800

92. Toil Impacts Development As Well
“28% - 63% of development teams’ total
time was consumed by operations toil.”
• 2016-2017 study of development teams
• 14 enterprises (insurance, healthcare, finance, travel, retail)
• Tech org headcount: 900 - 6800

93. Toil Impacts Development As Well
“28% - 63% of development teams’ total
time was consumed by operations toil.”
Waiting for environments Incident escalations
Rework due to env. differences Handoffs
Network issues Requests for information
Broken lower environments Change meetings
And more…
• 2016-2017 study of development teams
• 14 enterprises (insurance, healthcare, finance, travel, retail)
• Tech org headcount: 900 - 6800

94. Forces That Undermine Operations
Silos Queues
Toil Low Trust

95. Where are decisions made? Who can take action?
escalate
1° 2° 3° 4°
escalate escalateor

96. All work is contextual
John
Allspaw

97. All work is contextual
rm -rf $PATHNAME
John
Allspaw

98. All work is contextual
rm -rf $PATHNAME Is this dangerous?
John
Allspaw

99. All work is contextual
rm -rf $PATHNAME
John
Allspaw

100. All work is contextual
rm -rf $PATHNAME
John
Allspaw

101. All work is contextual
rm -rf $PATHNAME
Is this dangerous?
John
Allspaw

102. All work is contextual
rm -rf $PATHNAME
John
Allspaw

103. All work is contextual
rm -rf $PATHNAME
Answer is always
“it depends”
John
Allspaw

104. escalate
1° 2° 3° 4°
escalate escalateor
Context
Where are decisions made? Who can take action?

105. Low trust + approvals = illusion of control
Ticket
System

106. Low trust + approvals = illusion of control
Ticket
System
Add up the total number of approval requests and

107. Low trust + approvals = illusion of control
Ticket
System
Add up the total number of approval requests and
…subtract the info radiators (“I need to be in the loop”)

108. Low trust + approvals = illusion of control
Ticket
System
Add up the total number of approval requests and
…subtract the info radiators (“I need to be in the loop”)
…subtract the CYAs (“Prove you followed the process”)

109. Low trust + approvals = illusion of control
Ticket
System
Add up the total number of approval requests and
…subtract the info radiators (“I need to be in the loop”)
…subtract the CYAs (“Prove you followed the process”)
…subtract the too removed to judge (“mostly guessing”)

110. Low trust + approvals = illusion of control
Ticket
System
Add up the total number of approval requests and
…subtract the info radiators (“I need to be in the loop”)
…subtract the CYAs (“Prove you followed the process”)
…subtract the too removed to judge (“mostly guessing”)

111. Low trust + approvals = illusion of control
Ticket
System
Add up the total number of approval requests and
…subtract the info radiators (“I need to be in the loop”)
…subtract the CYAs (“Prove you followed the process”)
…subtract the too removed to judge (“mostly guessing”)

112. Low trust + approvals = illusion of control
Ticket
System
Add up the total number of approval requests and
…subtract the info radiators (“I need to be in the loop”)
…subtract the CYAs (“Prove you followed the process”)
…subtract the too removed to judge (“mostly guessing”)
How many are you left with?

113. Low trust + approvals = illusion of control
Ticket
System
Add up the total number of approval requests and
…subtract the info radiators (“I need to be in the loop”)
…subtract the CYAs (“Prove you followed the process”)
…subtract the too removed to judge (“mostly guessing”)
How many are you left with?
How many were the right call?

114. Low trust + approvals = illusion of control
Ticket
System
Add up the total number of approval requests and
…subtract the info radiators (“I need to be in the loop”)
…subtract the CYAs (“Prove you followed the process”)
…subtract the too removed to judge (“mostly guessing”)
How many are you left with?
How many were the right call?
How many got rejected?

115. Low trust + approvals = illusion of control
Ticket
System
Add up the total number of approval requests and
…subtract the info radiators (“I need to be in the loop”)
…subtract the CYAs (“Prove you followed the process”)
…subtract the too removed to judge (“mostly guessing”)
How many are you left with?
How many were the right call?
How many got rejected?

116. Forces That Undermine Operations
Silos Queues
Toil Low Trust

117. So what can we do differently?

118. Obvious: Get rid of as many silos as possible
Old Silo A Old Silo B Old Silo C Old Silo D

119. Old Silo A Old Silo B Old Silo C Old Silo D
Cross-Functional Team 1
Cross-Functional Team 2
Cross-Functional Team n
Obvious: Get rid of as many silos as possible

120. Old Silo A Old Silo B Old Silo C Old Silo D
Cross-Functional Team 1
Cross-Functional Team 2
Cross-Functional Team n
Obvious: Get rid of as many silos as possible
“Horizontal” shared
responsibility, not
everyone do everything!

121. Shared and dedicated responsibility is key
Cross-Functional Team 1
Cross-Functional Team 2
Cross-Functional Team n
Development Team 1
Development Team 2
Development Team n
SRE
Team
Clear handoff requirements
Error budget consequences
“Netflix"
Model
“Google”
Model

122. Shared and dedicated responsibility is key
Cross-Functional Team 1
Cross-Functional Team 2
Cross-Functional Team n
Development Team 1
Development Team 2
Development Team n
SRE
Team
Clear handoff requirements
Error budget consequences
“Netflix"
Model
“Google”
Model

123. Shared and dedicated responsibility is key
Cross-Functional Team 1
Cross-Functional Team 2
Cross-Functional Team n
Development Team 1
Development Team 2
Development Team n
SRE
Team
Clear handoff requirements
Error budget consequences
“Netflix"
Model
“Google”
Model
Same
high-quality,
high-velocity
results!

124. But what about the cross-cutting concerns?
Old Silo A Old Silo B Old Silo C Old Silo D
Cross-Functional Team 1
Cross-Functional Team 2
Cross-Functional Team n
Specialist
Capabilities
Specialist
Capabilities
Specialist
Capabilities

125. But what about the cross-cutting concerns?
Old Silo A Old Silo B Old Silo C Old Silo D
Cross-Functional Team 1
Cross-Functional Team 2
Cross-Functional Team n
Specialist
Capabilities
Specialist
Capabilities
Specialist
Capabilities
Ticket
Queue
Ticket
Queue
Ticket
Queue

126. But what about the cross-cutting concerns?
Old Silo A Old Silo B Old Silo C Old Silo D
Cross-Functional Team 1
Cross-Functional Team 2
Cross-Functional Team n
Specialist
Capabilities
Specialist
Capabilities
Specialist
Capabilities
Ticket
Queue
Ticket
Queue
Ticket
Queue
Ticket
Queue
Ticket
Queue Ticket
Queue

127. Operations as a Service: Turn handoffs into self-service
Operations as a Service
On
Demand
On
Demand
On
Demand
On
Demand
Ops
(embedded)Cross-Functional Product Team 1
Cross-Functional Product Team n Ops
(embedded)
Ops
(builds & operates)
Cross-Functional Product Team 2 Ops
(embedded)
Ops Capability
SRE, Dev, or
Specialist
Ops Capability
SRE, Dev, or
Specialist
Ops Capability
SRE, Dev, or
Specialist

128. Development Team 1
Development Team 2
Development Team n
Ops/SRE
Team
Operations as a Service
On
Demand
On
Demand
On
Demand
On
Demand
Ops
(builds & operates)
Ops Capability
SRE, Dev, or
Specialist
Ops Capability
SRE, Dev, or
Specialist
Ops Capability
SRE, Dev, or
Specialist
Operations as a Service: Works with any org model

129. Use tickets only for what they are good for
Ticket
System

130. Use tickets only for what they are good for
1.Documenting true problems/issues/exceptions
Ticket
System

131. Use tickets only for what they are good for
1.Documenting true problems/issues/exceptions
2.Routing for necessary approvals
Ticket
System

132. Use tickets only for what they are good for
1.Documenting true problems/issues/exceptions
2.Routing for necessary approvals
Not as a general purpose work management system!
Ticket
System

133. Security or compliance “in the way”?
Operations as a Service
On
Demand
On
Demand
On
Demand
On
Demand
Ops
(embedded)Cross-Functional Product Team 1
Cross-Functional Product Team n Ops
(embedded)
Ops
(builds & operates)
Cross-Functional Product Team 2 Ops
(embedded)
Ops Capability
SRE, Dev, or
Specialist
Ops Capability
SRE, Dev, or
Specialist
Ops Capability
SRE, Dev, or
Specialist
Build-in
Security
Here
Build-in
Compliance
Here

134. “Shift Left” the ability to take action
escalate
1° 2° 3° 4°
escalate escalateor

135. “Shift Left” the ability to take action
Push the ability to take action this direction
escalate
1° 2° 3° 4°
escalate escalateor

136. “Shift Left” the ability to take action
Push the ability to take action this direction
escalate
1° 2° 3° 4°
escalate escalateor
OaaS Enablement and tooling

137. Reduce Toil

138. Reduce Toil
1. Track toil levels for each team

139. Reduce Toil
1. Track toil levels for each team
2. Set toil limits for each team

140. Reduce Toil
1. Track toil levels for each team
2. Set toil limits for each team
3. Fund efforts to reduce toil (with emphasis on teams over toil limits)

141. Reduce Toil
1. Track toil levels for each team
2. Set toil limits for each team
3. Fund efforts to reduce toil (with emphasis on teams over toil limits)
Bonus: Use Service Level Objectives, Error Budgets, and other lessons from SRE

142. Example Operations as a Service Platform (shameless plug)
#! Ȑ Ƙ
Scripts APIs Tools Cloud VMs Containers
Workflow and
Scheduling
Collect and
Process Output
Infrastructure
details and state
Config.
Man.
CMDB
Monitor.
Metrics
Cloud
Corp
Directory
Authentication
and roles
ITSM Tickets, work
status, approvals
>_
Create workflows ● Define policies ● Execute workflows
Web GUI API CLI

143. Recap
Don’t forget about Ops.
Challenge conventional wisdom.
Leverage the Operations as a
Service design pattern
“Shift-Left” control and decision
making.
Old Silo A Old Silo B Old Silo C Old Silo D
Cross-Functional Team 1
Cross-Functional Team 2
Cross-Functional Team n
Focus on removing silos and
queues
Operations as a Service
On
Demand
On
Demand
On
Demand
On
Demand
Ops
(embedded)Cross-Functional Product Team 1
Cross-Functional Product Team n Ops
(embedded)
Ops
(builds & operates)
Cross-Functional Product Team 2 Ops
(embedded)
Ops Capability
SRE, Dev, or
Specialist
Ops Capability
SRE, Dev, or
Specialist
Ops Capability
SRE, Dev, or
Specialist
Learn from SRE: Reduce toil to
create capacity to change
Toil Engineering Work
E.W.Toil
Reduce toil
Improve the business ǡ
No capacity to reduce toil
No capacity to improve business
Toil at manageable percentage of capacity
Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”)
Understand the forces
undermining operations work

144. Let’s talk…
@damonedwards
damon@rundeck.com
https://www.rundeck.com/oaas
Dive Deeper Into Operations as a Service: