Three pillars of observability

1. Observability:
- Logs
- Metrics
- Traces

2. Outline
1. Observability
2. Service Level Terminology
3. Three Pillars of Observability
a. Logs
b. Metrics
c. Traces
4. Demo

4. Monitoring
- What’s broken & why?
- Types of monitoring:
a. Black box
b. White box

7. Observability
- A measure of how well the internal states of a system can be inferred from
knowledge of its external outputs
- Provides context

8. Quote
Hope is not a strategy.
Traditional SRE saying

9. Service Level Terminology

10. Four Golden Signals by Google SRE

11. Apdex - Define T

12. Apdex - Calculate

13. Apdex - Evaluate

14. Apdex - Visualization

15. Three Pillars of Observability?

16. 1. Logs
- Aggregate them to a central location
- Transform
- Index properly
- Visualize

17. Elastic Stack

19. Elastic Stack - Best Practices
- Do not lose log data
- Monitor logstash/elasticsearch
- Data retention
- Test and verify configurations
- Log request/response models

20. 2. Metrics

21. Prometheus

23. Spring Prometheus Actuator

24. Prometheus - Best Practices
- Use recording rules to create sub metrics
- Write unit tests for alert rules
- Alertmanager deployment
- At least two instances of alertmanager
- Do not use load balancer
- Send alerts to all alertmanager instances
- Unsee by cloudflare

25. Track Every Release

26. 3. Traces

27. Distributed Tracing Model

29. Jaeger
- OpenTracing compatible data model and instrumentation libraries
- Features
- Distributed transaction monitoring
- Root cause analysis
- Service dependency analysis

30. Trace

31. Span Detail

32. Sampling
- Constant
- Probabilistic
- Rate Limiting
- Remote: central configuration, adaptive sampling

34. Demo

35. References
- Cindy Siridharan: https://medium.com/@copyconstruct/monitoring-and-observability-
8417d1952e1c
- Google SRE Book: https://landing.google.com/sre/sre-book/toc/
- Measure Anything, Measure Everything: https://codeascraft.com/2011/02/15/measure-
anything-measure-everything/
- Elastic Stack: https://logz.io/learn/complete-guide-elk-stack
- Observability with Elastic Stack: https://www.elastic.co/blog/observability-with-the-
elastic-stack
- Track Every Release: https://codeascraft.com/2010/12/08/track-every-release/
- Monitoring at Peak, Java Day Istanbul 2019 :
https://www.youtube.com/watch?v=xhPx_WFFn5I
- Promtool Docker: https://github.com/dnanexus/docker-promtool
- Apdex: http://apdex.org/overview.html
- Demo: https://github.com/enesaltinok/observability

36. Questions?

37. Thank you…
Enes Altınok
enes.altinok@emlakjet.com