Observability für alle

Observability für alle
Cloud Native Night
25. Oktober 2018
Florian Lautenschlager
florian.lautenschlager@qaware.de
@flolaut
Josef Fuchshuber
josef.fuchshuber@qaware.de
@fuchshuber

Observability für alle 3
In our cloud backend we have a vital microservice ecosystem.

Our team is just as vital and heterogeneous as our software.
Platform Developer
App Developer
Skill Developer
Client Developer
Tester
Ops
Help Desk
Product Management
Data Scientist
UX Designer

Observability isn't just for operations.

What is the hardest step in the DevOps process?
DEV OPS

Much better: The 6 Cs of the DevOps Cycle.
Source: https://dzone.com/articles/6-cs-of-devops-adoption

Observability in the wild!
A case study… and how we found
collaborative monitoring.

Monitoring Toolchain: Simply Cloud Native.
Metrics Events Traces
Java (Spring Boot) or Python
on
Azure / Kubernetes / Openshift / Docker

Monitoring
Technical and Functional
Kubernetes
Generic monitoring that
does not need knowledge
about the application.
Monitoring that does
need knowledge about
the application.
Health of platform and application Telemetry data
Infrastructure-Monitoring
Application-Monitoring

Monitoring
Technical and Functional
Questions:
Services are up and running
Services can accept traffic
Sources:
Kubestate-Exporter
Prometheus-Node-Exporter
JMX, top, iostat etc.
Questions:
Use-Cases runtimes
Service level agreements
Sources:
Specific instrumentation
(around use cases, etc.)
Health of platform and application Telemetry data
Kubernetes
Infrastructure-Monitoring
Application-Monitoring

Total duration
Involved services <click>

Code-Slide: Standardize tracing logs and tags.
Span logs: We model database calls as well
as other expensive calls as logs using a
template to reduce the size of traces:
db:<Repo>.<Call> took: xx ms.
call:<Class>.<Method> took: xx ms.
Span tags: Used to model values that are
valid for a span. We use a template to
standardize tags.
span.tag. (to mark our tags)
Environment (staging, integration , etc.)
db (to mark spans with db calls.)
param.<name>=value (call parameters)

Logs for
a given
trace
Involved
Services

Code-Slide: Contextual logging.
Context of a log event.
Everyone can easily see the logs for a specific context (trace etc.)

end-2-end tests are also integrated in our observability stack.
17
See the logs

We provide these tools and techniques to every
developer, but…
Best SmartSpeaker in the World. Best Software-System in the World.Best Developers in the World.
Blah blah Weather blah
(voice)
Don’t understand “Blah blah Weather blah”
… in case of an error the experts of the best software
system in the world were often asked, what is the
problem?

I know. Most of you do this already. But what about ..
Collaborative Monitoring!?!?

An example is the best explanation.
and a chatbot…
and a monitoring toolchain…
Once there was a little tiny application…

Snip Snap
Links request
with trace and
logs.
verbose

Or in case
of an error

Or for checking
the health of
the services

Or for checking
the status of
e2e tests

Our current setup: A chatbot as generic interface.

Observability für alle
Happy end.
27

Summary
Collaborative Monitoring:
Monitoring that allows everyone to benefit
of without the need of expert knowledge.

Three steps to enable collaborative monitoring.
29
Standardize
metrics, logs
and traces
Link and
combine them
as far as
possible
Integrate them
into everyone's
tools
Start Here
Correlate Events and Trace by Context
Metrics with Events and Traces by Time
Structured Logging + Context, Metric names, etc.
Tools your team

Did we create an uncontrollable observability monster?

There’s No
Such Thing as
a Free Lunch
• The more complex a
microservice architecture is,
the more sophisticated the
observability solution must be.
• For Collaborative Observability
there is no out of the box
solution.

Collaborative Monitoring by everyone.
Ease of use.
Simple general interface to access various monitoring tools.
Integrated into everyone's daily tools (ChatBots, E-Mail, etc.)
Support all kinds of teams: Operations / Dev-Ops / Developers / QA-Team / My mum =)
Allow everyone to get superman insights.
Decrease Mean Time To Recovery (MTTR) with a fast analysis
Integrates different kinds of monitoring data (traces, metrics and logs) of different monitoring layers.
The right information. Provide relevant information for different teams, e.g. runtimes for perf. engineer.
Level of Detail: Abstract (use case level) for management vs. details (database calls) for developers

Ease of communication
within bug tickets.

Lessons Learned
Tool stack is awesome: Prometheus, Sleuth / Zipkin, Logging (fluentD, elastic) is stable with a good
documentation.
Maximum flexibility compared to commercial products.
But: Effort for concepts, implementation and quality checks. Conventions and rulesets are important!
Mindset:We found that we had to convince people first. But we have seen a high level of acceptance.
Example: Chatbot with trace-links is standard tool for discussing possible bugs between all project roles.
Development and system understanding: No need of “cloudy” conversations. Just provide the context,
e.g. a trace id.
Example: Issues typically contain the context (trace id) that points the developer to the logs and the trace.
Mark customer and automatic test traffic for better dashboards and analytics.
Observability tool stack is a first calls citizen:You do not make friends when it's down

Observability für alle

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (12)

Semelhante a Observability für alle

Semelhante a Observability für alle (20)

Mais de QAware GmbH

Mais de QAware GmbH (20)

Último

Último (20)

Observability für alle