Observability - Experiencing the “why” behind the jargon (FlowCon 2019)

@A_Bangser @FlowConFR #FlowCon
My slides are / will be available for you at:
Observability -
Experiencing the “why” behind the jargon
Abby Bangser
https://www.slideshare.net/AbigailBangser

Observability
In control theory, observability is a measure of how well
internal states of a system can be inferred from knowledge of
its external outputs.

“measure of how well” means observability is a scale

Incident
triage
Incident
triage
happening?!

How easy is it to answer a new question without deploying new code?
Incident
triage
Incident
triage
happening?!
observability observability
observability

Observability
In control theory, observability is a measure of how well
internal states of a system can be inferred from knowledge
of its external outputs.

External outputs help us answer these questions

So you might be thinking… “right, monitoring”
https://bravenewgeek.com/wp-content/uploads/2019/10/monitoring_vs_observability_overlay-1024x539.png

True observability is discovering new behaviours
https://bravenewgeek.com/wp-content/uploads/2019/10/monitoring_vs_observability_overlay-1024x539.png

Characteristics of what generates valuable outputs
https://thenewstack.io/observability-a-3-year-retrospective/
➔ raw events
➔ no pre-aggregation
➔ structured data
➔ arbitrarily wide events
➔ schema-less-ness
➔ high cardinality dimensions
➔ oriented around the lifecycle of the request
➔ batched up context
➔ static dashboards don’t work, it must be exploratory

Characteristics of what generates valuable outputs
https://thenewstack.io/observability-a-3-year-retrospective/
➔ raw events
➔ structured data
ByTwitter,CCBY4.0,
https://commons.wikimedia.org/w/index.php?curid=76921548

Let’s understand a couple of these through examples
➔ raw events
➔ structured data

The promise of monitoring vs my reality
My rollercoaster journey with understanding metrics and
pre-aggregation starts back in 2016...

Monitorama 2016 - an awakening
Lessons include…
➔ It is not just testing that is dead
➔ Wow! There is a world of available data I have no idea about
➔ These tools are so cool...wait, what are these tools?

Metrics can track success (and failure) of changes made
https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/#xref_monitoring_golden-signals

An ask:
I want to monitor live
systems
An opportunity:
Help create a
client’s ﬁrst cloud
infrastructure

An operations focused project changed my tool chain

@A_Bangser @FlowConFR #FlowCon@A_Bangser @FlowConFR #FlowCon
And then…
just like testability,
operability became
hard to prioritise

Two years and many projects later Hobbsy had a plan
Track latency over 4 weeks and alert when current trends exceed 2 standard deviations
2standarddeviations

To do this at MOO
s / MOO / any company over a few years old /
➔ 40 services
➔ 4 core languages
➔ 3 eras of architectural decisions
➔ 2 transport protocols (http and gRPC)

To do this at MOO
s / MOO / any company over a few years old /
➔ 40 services
➔ 4 core languages
➔ 3 eras of architectural decisions
➔ 2 transport protocols (http and gRPC)
...and a partridge in a pear tree

The plan: Standardise metrics across the estate

Consistency across services created so much learning

But...

Our data collection made certain assumptions which
in the end required re-collecting in a different way

How histograms gets generated in a time series DB
le= 0.05
http_requests_seconds_bucket
le= 0.1 le= 0.5 le= 1 le= 5 le= +inf
* “le” stands for “less than or equal to”

le= 0.05
le= 0.1 le= 0.5 le= 1 le= 5 le= +inf
www.moo.com in 0.25 seconds

le= 0.05
le= 0.1 le= 0.5 le= 1 le= 5 le= +inf
www.moo.com/big_ﬁle in 5 seconds
www.moo.com in 0.25 seconds

We collected counts of how many requests per bucket
le= .05
le= .1 le= .5 le= 1 le= 5 le= +inf
Offset
1week2week3week4week
le= .05 le= .1 le= .5 le= 1 le= 5 le= +inf

The data we had collected, we had to throw away
Offset
1week2week3week4week
le= .05 le= 5le= .5le= .1 le= +infle= 1
le= .05 le= .1 le= +infle= 1
le= 5le= .1 le= +infle= 1
le= .05 le= 5le= .1 le= .5
le= .05
le= .5
le= .5
le= 1
le= 5
le= +inf

At least the update was made, now we are all set right?

le= .05 le= 5le= .1 le= .5 le= 1 le= +inf
Except, that 99th percentile...what does that actually mean?

Let’s see what our logs say about it

Just 1% of 500,000 requests applies to 56,000 people

To see >10 seconds, I would need the 99.996%

So, while consistent metrics
trending over time was a big
step forward...
In retrospect,
these experiences were
not mature observability

Why avoid pre-aggregation?
Because you can never regain the original context and detail,
you can only ever ask predetermined questions

Data is not the same as information
Step one is accepting that while sentences may be readable.
<key : value> pairs are more easily queried.

Even from the ﬁrst “Hello World” we humans logged

And from there we wanted more information
7a82dd3a

So then we backﬁlled in structure
grok {
match => [
"Request",
"%{URIPROTO:request_uri_scheme}://
%{HOSTNAME:request_uri_host}(?::%{POSINT:request_uri_port})
?%{URIPATH:request_uri_path}(?:%{URIPARAM:request_uri_query})?"
]}
}

And of course, from there we wanted more
mutate {
split => { "uri_array" => "/"}
add_field => {
"uri_root" => ["/%{[uri_array][1]}"]
"uri_first" => ["/%{[uri_array][2]}"]
"uri_second" => ["/%{[uri_array][3]}"]
"uri_root_first" => "%{uri_root}%{uri_first}"
"uri_root_second" => "%{uri_root}%{uri_first}%{uri_second}"
}

And even looking past the bad ﬁelds values, lots of
servers means lots of intermingled logs

Rewind...how are logs written during a request?

Detailing how logs get written during a request
@PostMapping("flip")
public ResponseEntity flipImage(@RequestParam("image") MultipartFile file,
@RequestParam(value = "vertical") Boolean vertical,
@RequestParam(value = "horizontal") Boolean horizontal) {
if (file.getContentType() != null) {
LOGGER.warn("Wrong content type uploaded: {}", file.getContentType());
return new ResponseEntity<>("Wrong content type uploaded: " + file.getContentType());
}
LOGGER.info("Receiving {} image to flip.", file.getContentType());
byte[] flippedImage = imageService.flip(file, vertical, horizontal);
if (flippedImage == null) {
return new ResponseEntity<>("Failed to flip image", HttpStatus.INTERNAL_SERVER_ERROR);
}
LOGGER.info("Successfully flipped image id: {}", file.getId());
return new ResponseEntity<>(flippedImage, headers, HttpStatus.OK);
}

}
}
}

}
}
}
LOGGER.info("Successfully flipped image id: {}", file.getId()");

Log outputs

In contrast, how an event is created during a request
public ResponseEntity flipImage(...) {
EVENT.addField("content.type", file.getContentType());
EVENT.addField("action", "flip");
EVENT.addField("image_id", file.getId());
EVENT.addField("flip_vertical", vertical);
EVENT.addField("flip_horizontal", horizontal);
...
...
EVENT.addField("action.success", "true");
}

...
...
}

Comparing the outputs
Multiple logs
A single event

Making the information easy to query

And keeping the information in context

Most importantly, making it easy to add more!

In order to combate tribal knowledge based guessing
when debugging our complex systems, we need:
A low friction way to add ﬁelds to your
logs for structure and searchability
Allowing application and user context to
be wrapped in a business context
CustomerID:234567VersionOfApp:2
RequestedUri:www.

Debugging distributed systems is hard
Especially when business impact is on the line.
Let’s talk outages

Hmmm, a warning alert has come in
This is an automated alert based on a warning production service sending a high percent of 500’s in production!

Yup, deﬁnitely an issue

All hands on deck, what is happening...and why?

2+ hrs and still aren’t sure we know what happened

And then it keeps happening

Oncall engineers are not amused

But the service owners weren’t just lounging around

And these were some awesome dashboards

Let’s break down what this dashboard shows
Request Counts Response Latency

Let’s break down what this dashboard shows
Enhanced Images
Original Images
Enhanced Images
Enhanced and resized
Request Counts Response Latency

This dashboard helped limit impact
~3 hours
40 min

And eventually, powerful human pattern matchers
solved the problem

So what happens to this dashboard now?

They have been sent to a farm… with their other friends

Why ditch the dashboards?
The scar tissue of your past outages is not a sufﬁcient
replacement for the creativity required to investigate your
future incidents
https://www.needpix.com/photo/907639/images-leash-leash-polaroid-free-pictures-free-photos-free-images-royalty-free

Let’s revisit those characteristics
➔ raw events
➔ structured data
➔ static dashboards don’t work, it must be
exploratory
ByTwitter,CCBY4.0,https://commons.wikimedia.org/w/index.php?curid=80936515

➔ raw events
➔ structured data
exploratory
The only way to ask new questions
is to keep the original raw data
available and queryable

➔ raw events
➔ structured data
exploratory
Make data easy to
add details to and
easy to query

➔ raw events
➔ structured data
exploratory
Empower creative
and shared
exploration based
on business context

➔ raw events
➔ structured data
exploratory
The only way to ask new questions
is to keep the original raw data
available and queryable
Make data easy to
add details to and
easy to query
Empower creative
and shared
exploration based
on business context

QA
TWU
Looking back journeys are never clear, so why do we
still expect them to be when we start a new one?
Political
Science Major
Data analysis for
investments
A desire to
learn how to
code
Automation
FTW!
An “analyst”
computer
A “DevOps”
friend
engaged me
in his work
onitorama
An infrastructure
project
Platform
Engineering @
Professional
scuba diver
A (slight)
obsession with
observability

Start where you are.
Use what you have.
Do what you can.
- Arthur Ashe

➔ All of tech and product is now asking more interesting questions
➔ We are expecting more of our tooling
➔ We are building new awareness about our services and system
Start where you are.
Use what you have.
Do what you can.
- Arthur Ashe

Thank you!
www.SlideShare.net/
AbigailBangser
@A_Bangser

Observability - Experiencing the “why” behind the jargon (FlowCon 2019)

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Observability - Experiencing the “why” behind the jargon (FlowCon 2019)

Semelhante a Observability - Experiencing the “why” behind the jargon (FlowCon 2019) (20)

Mais de Abigail Bangser

Mais de Abigail Bangser (10)

Último

Último (20)

Observability - Experiencing the “why” behind the jargon (FlowCon 2019)