The servicescore card - Gamifying Operational Excellence - SRECON

Gamifying Operational Excellence
The
Service
Score
Card

1 The Problem
3 A Solution tour
4 The results
5 Take aways & lessons Learnt & Questions
2 A Solution idea
Agenda

“If it's not broken, I’ll fix it.”
From Australia, on loan as
Staff SRE @ linkedIn
jobs, companies, recruiter
& Finder of encoding bugs
about me
Danny ☃ Lawrence

Good news
SRECON.
You passed
the ☃ test.
about me
Danny ☃ Lawrence

Some terms
(before we really get started)

Operational Excellence
effective and efficient delivery of information,
technology, and services required by end users
that add measurable value.
10

Operational Excellence
Doing everything required to make sure
all of your services are as fast and as reliable
as possible.
11

Gamification
application of game-design elements and game
principles in non-game contexts.
12

Some background
(LinkedIn SRE crash course)

Mostly Java
Multitudes of services
Doing lots of things
Service-oriented architecture
Everything talks to everything
My direct team looks after 80+ services
We have 200+ SREs
14
LinkedIn SRE Crash Course

The Problem
(What started this whole thing)

Problem 1:
The GOOD
&
The BAD
16

BAD services
wake me up
17

GOOD services
let me sleep
18

What makes a GOOD
service at LinkedIn
is a moving target.
19

Technologies and dependencies
change
over time.
20

Upgrading dependencies & libraries
Java / Jetty / Play / Tomcat
Correct usage of TLS
Switching databases / caches
Migrate from SVN to GIT
Reduce application startup time
Setup error budgeting
True up the number of metrics
21
Some examples

A GOOD service
can turn into a BAD service.
If you are not checking it
22

Unfortunately
BAD services
do not
magically
turn into
GOOD services
23

Problem 2:
Knowing what is BAD
24

Problem 3:
Knowing why it’s BAD
25

Problem 4:
Tribal knowledge
about how to get to GOOD
26

The only thing SREs hate more than
not having documentation.
Is writing documentation.
27

BAD services wake me up
Time will cause GOOD to turn BAD
Hard to know what is BAD
Hard to know why is BAD
Not sure how to fix the BAD
29

The Service ScoreCard
(A solution)

In order determine the health
of the services we support,
we define a list of production requirements.
31

Apply a weight to each requirement
32

Codify each requirement into a check.
33

Execute these checks
for each service
34
Service Scorecard

Tally up the results for service.
35

Grade the service from
“F” to “A+”
36

Add all the services into a highscore system
37

Then
38

Publish those scores to the company
39

This is great,
but how do I improve the score?
How can I add X check into the system.
40

checks are one type of plugin.
fetch plugins gather data
check plugins check the data.
42

We use the fetch plugin to gather
remote data from:
SVN, GIT, Configuration DBs,
host databases, monitoring systems,
build systems, deployment systems.
43

Basically,
if we can fetch it,
then we do so.
44

We build a giant context object.
45

The check plugin will look at our
context object.
46

All plugins are small python scripts,
where small is 10~30 LOC
47

Simply return 2 or 3 things.
state*: True, False, None or 0.0 - 1.0
message*: short string
data: python dict of interesting things.
48

@ssc.tags(“ownership”)
def fetch_ownership(service_name):
“Fetch all the ownership data of a service”
o = r.get(“http://owners/” + service_name)
return True, “gathered data”, o.json()
50

51

52

53

return True, “gathered owner data”, o.json()
54

@ssc.weight(5)
@ssc.tags(‘ownership’)
@ssc.wiki(‘http://wiki/ssc_eng_owner’)
def check_eng_team(ctx):
“ensure ENG ownership of a service”
if ctx.ownership.eng_team:
return True, ctx.ownership.eng_team
return False, “missing eng_team”
56

@ssc.weight(5)
57

@ssc.weight(5)
58

@ssc.weight(5)
59

@ssc.weight(5)
60

@ssc.weight(5)
61

@ssc.weight(5)
62

Problems
Understanding what is BAD
Knowing why it is BAD
64

Problems
65

Problems
80

Problems
94

What is the check?
Why is it important?
How long it will take to fix?
How will it be fixed?
100

102
AngularJS
image: CC BY 4.0 https://angular.io/presskit.html (2017)

103
{{service_name}}
becomes
jobs-server

105
{{context.ownership.eng_owner}}
becomes
jobs-team

Using our fetched data in the wiki

108
{html}
<script src=”https://cdn/angularjs.js”/ >
{html}

109
var query = $location.search();
var service_name = query[‘service_name’];
var url = ‘http://ssc/api/’ + service_name;
$http.get().success(
function(ctx) {
$scope.ctx = ctx;
}
);

110
function(ctx) {
$scope.ctx = ctx;
}
);

111
function(ctx) {
$scope.ctx = ctx;
}
);

112
function(ctx) {
$scope.ctx = ctx;
}
);

113
function(ctx) {
$scope.ctx = data;
}
);

114
function(ctx) {
$scope.ctx = ctx;
}
);

115
{{ctx.ownership.owner_eng}}

116
{{ctx.ownership.owner_eng}}
{{ctx.number_of_hosts}}
{{ctx.product.lib.jetty.version}}
{{ctx.hosts.hostnames}}
{{ctx.is_deployed_in_prod}}
{{ctx.commits.last_commit}}

Problems
117

Now
Reports show what is BAD
Checks validate why it is BAD
Wiki shows how to fix the BAD
118

No more of these emails
“If you use a lib-core, then upgrade it,
we found a bug”
119

How many of my 80 services use this lib?
How do I check?
How do I upgrade?
120

125
pre-commit Build Deployment Monitoring

126
Service Scorecard

127
Service Scorecard
API

128
Service Scorecard
API
hack-days Reporting Deployment Monitoring

What we do with the scores?
130

Priority #1:
Getting the grades better
131

132
When we started Now
Average grade for my team 40% 80%
Average score across SRE 35% 60%
Checks in 24 hours 15,560 89,859
Number of checks per service 15 31

We can now explore news ways
to use the scores
133

Carrot
&
Stick
134

Carrot / GOOD
Stick / BAD
135

No SRE support
for
F Grade
services.
136

F Grade services generally
cause the
most problems.
137

No deploy moratorium
for
A+ services
138

A+ services generally
cause the
least problems.
139

A services
are allowed to deploy 24/7
140

Premium SRE support
for A+ services
141

Priority build queues
for
GOOD
Services.
142

Tiger teams
to raise the
scores on
F Grade services
143

Hack Days
144

FREE BEER
145

Basically any problem
can be solve with
FREE BEER
146

OR T-Shirts
147

Influence where we allocate
open headcount
149

Simple way to get things done
150

Everyone cares about Reliability.
152

Everyone cares about Reliability,
Everyone is a Site Reliability Engineer.
153

Everyone cares about Reliability,
You just need to empower them.
154

Hack Days are important,
This POC was built in an afternoon.
155

Getting the data was easy,
Finding interesting ways to use it is hard.
156

Make it as easy as possible
to do the right thing.
157

The servicescore card - Gamifying Operational Excellence - SRECON

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a The servicescore card - Gamifying Operational Excellence - SRECON

Semelhante a The servicescore card - Gamifying Operational Excellence - SRECON (20)

Último

Último (20)

The servicescore card - Gamifying Operational Excellence - SRECON