This document discusses concepts related to observability including Prometheus, ELK stack, OpenTracing, and Victoria Metrics. It provides examples of setting up Prometheus and Grafana to monitor metrics from applications instrumented with exporters. It also demonstrates setting up Filebeat, Logstash and Elasticsearch (ELK stack) to monitor logs and send them to Elasticsearch. Additionally, it shows how to implement OpenTracing in a Java application and visualize traces using Jaeger. Finally, it outlines an exercise to build a microservices ecommerce application incorporating logging, metrics and tracing using the discussed tools.

1. Observability
Diego Pacheco

3. Observability

4. Observability

5. Observability

6. Observability

7. Observability

8. Observability

9. Observability

10. Observability

11. Observability

12. Observability

13. Observability

14. Observability

15. Observability

16. Observability

17. Observability - Prometheus Architecture

18. Observability

19. Observability

20. Observability

21. Observability

22. Observability

23. ELK
https://elk-docker.readthedocs.io/https://www.elastic.co/what-is/elk-stack

24. ELK

25. ELK

26. ELK

27. ELK: Create a document and query

28. ELK: Query ES

29. ELK: Setup Index and add documents

30. ELK: Setup Index Pattern in Kibana and them query

31. ELK: Filebeat + Redis
https://gist.github.com/diegopacheco/e35fb0adceb34b54c4dc4d304d686cd5

32. ELK: Filebeat + Redis: Enable slowlog tracing

33. ELK: Filebeat + Redis: Enable slowlog tracing

34. ELK: Filebeat + Redis
https://gist.github.com/diegopacheco/e35fb0adceb34b54c4dc4d304d686cd5

35. ELK: Filebeat + Redis: Documentation inside ELK

36. ELK: Filebeat + Redis: Check if is all working!

37. ELK: Filebeat + Redis: Data in ELK

38. ELK: Java Application ~ Spring Boot 2x + Actuator + Micrometer

39. ELK: Java Application ~ Spring Boot 2x + Actuator + Micrometer
build.gradle
src/main/resources/application.properties

40. ELK: Java Application ~ Spring Boot 2x + Actuator + Micrometer
MainController.java

41. ELK: Java Application ~ Spring Boot 2x + Actuator + Micrometer
Application.java ->

42. ELK: Java Application ~ Spring Boot 2x + Actuator + Micrometer
$ ./gradlew bootRun

43. ELK: Java Application ~ Spring Boot 2x + Actuator + Micrometer
$ ./gradlew bootRun

44. ELK: Java Application ~ Spring Boot 2x + Actuator + Micrometer

45. ELK: Java Application ~ Spring Boot 2x + Actuator + Micrometer

46. ELK: Java Application ~ Sending Logs ~ Running Logstash
src/main/resources/logback.xml

47. ELK: Java Application ~ Sending Logs ~ Running Logstash
build.gradle

48. ELK: Java Application ~ Sending Logs ~ Running Logstash
build.gradle

49. ELK: Java Application ~ Sending Logs ~ Running Logstash
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.5.1.tar.gz
tar -zcvf logstash-7.5.1.tar.gz
rm -rf logstash-7.5.1.tar.gz
logback.conf

50. ELK: Java Application ~ Sending Logs ~ Running Logstash
bin/logstash -f $APP_PATH/logstash-conf/logback.conf

51. ELK: Java Application ~ Sending Logs ~ Running Logstash
You can see there is a new ES log - proof that it works. You can create a index on Kibana now.

52. ELK: Java Application ~ Sending Logs ~ Running Logstash

53. Observability

54. Observability

55. Observability

56. Observability

57. Observability

58. Observability

59. Observability
❏ Long Term Remote Storage for Prometheus
❏ Support prometheus query API
❏ Support for Grafana
❏ Outperform InfluxDB and TimescaleDB by 20x more performance on inserts / selects
❏ Uses 10x less ran than influxDB
❏ High Data Compression - 70x more compression than TimescaleDB
❏ Easy Operation (Easy backup to S3) Ingestion via: Prometheus, InfluxDB, Graphite APIs
❏ OSS Cluster

60. Observability
Benchmarks

61. docker run --network=host -it --rm -v
/tmp/victoria-metrics-data:/victoria-metrics-data -p 8428:8428
victoriametrics/victoria-metrics

62. Observability - curl -v http://127.0.0.1:8428/metrics

63. docker run --network=host --rm --name=grafana -p 3000:3000
grafana/grafana

64. docker run --network=host --rm --name=prometheus -v
$(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml -p
9090:9090 prom/prometheus

65. curl http://localhost:9090/metrics

66. Prometheus Redis Exporter
https://github.com/oliver006/redis_exporter

67. Prometheus Redis Exporter
docker run --network=host --rm --name=prometheus -v
$(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml -p
9090:9090 prom/prometheus

68. Prometheus UI

69. Grafana UI

70. Grafana UI

71. Prometheus Exporters
https://prometheus.io/docs/instrumenting/exporters/

72. Grafana Dashboard for Victoria Metrics
https://grafana.com/grafana/dashboards/10229
10229

73. Observability - Victoria Metrics Cluster

74. Open Tracing
https://opentracing.io/

75. Open Tracing - Java App ~ Run Jaeger
docker run --rm -it --network=host jaegertracing/all-in-one
http://localhost:16686/search

76. Open Tracing - Java App
build.gradle
application.properties

77. Open Tracing - Java App

78. Open Tracing - Java App

79. Exercises
Constraints
1. You can use Java, Scala or Clojure as Language
2. You can use SpringBoot2x, Jetty or Netty as Server
3. You need to use Logback | ELK | Jaeger
Let's build a simple ecommerce
1. Build 2 microservices, first microservice will store items and do stock control. The microservice need to
have 3 operations /add/$item where you should have a name, description, photoURL and units at stock
. Second endpoint need be able to retrieve an item by ID, last endpoint need to be able to update itens
on the stock /stock/$ID (increase or decrease items). Second service need provide SALEs so the sales
microservices will call items/stock microservices in order toto sales.
2. Add All logs in ELK
3. Add traces in Jaeger
4. Prodive custom metrics and add them in ELK create a dashboard showing: TOP sales, items that less
less, salesman who sells more, items with less units on stock.

80. Observability
Diego Pacheco