O slideshow foi denunciado.
Seu SlideShare está sendo baixado. ×

OSMC 2022 | Metrics Stream Processing Using Riemann by Pradeep Chhertri

Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio

Confira estes a seguir

1 de 38 Anúncio

OSMC 2022 | Metrics Stream Processing Using Riemann by Pradeep Chhertri

Baixar para ler offline

This talk will cover:

Introduction to Riemann
* In-memory stream processing system written in clojure
* How it can easily process millions of events per second
* Configuration is clojure code

What problems Riemann solves in the today’s world of Prometheus & Datadog
* High cardinality metrics problem: Since all processing is done in*memory, riemann can handle high cardinality issue much better than prometheus.
* Instant detection: Since riemann uses websockets, an issue is instantly reflected into dashboard. In prometheus, it will only get reflected after next scrape.

Riemann concepts
* Event
* Stream
* Index
* Integrations

Riemann Stream Processing Engine
* Types of functions on riemann streams.
* Examples:
* Combine multiple streams into one stream.
* Split one stream into multiple streams.
* Filter, roll up, throttle, coalesce events.

Extending Riemann
* How riemann schema is extendable and use for streaming other events like logs.

Demo
* Using docker-compose setup of riemann to present the power of riemann and riemann-dash.

This talk will cover:

Introduction to Riemann
* In-memory stream processing system written in clojure
* How it can easily process millions of events per second
* Configuration is clojure code

What problems Riemann solves in the today’s world of Prometheus & Datadog
* High cardinality metrics problem: Since all processing is done in*memory, riemann can handle high cardinality issue much better than prometheus.
* Instant detection: Since riemann uses websockets, an issue is instantly reflected into dashboard. In prometheus, it will only get reflected after next scrape.

Riemann concepts
* Event
* Stream
* Index
* Integrations

Riemann Stream Processing Engine
* Types of functions on riemann streams.
* Examples:
* Combine multiple streams into one stream.
* Split one stream into multiple streams.
* Filter, roll up, throttle, coalesce events.

Extending Riemann
* How riemann schema is extendable and use for streaming other events like logs.

Demo
* Using docker-compose setup of riemann to present the power of riemann and riemann-dash.

Anúncio
Anúncio

Mais Conteúdo rRelacionado

Mais recentes (20)

Anúncio

OSMC 2022 | Metrics Stream Processing Using Riemann by Pradeep Chhertri

  1. 1. Metrics Stream Processing Using Riemann Pradeep Chhetri
  2. 2. Metrics Stream Processing Using Riemann Pradeep Chhetri
  3. 3. Introduction
  4. 4. Introduction Event processing system designed specifically for metrics Written in Clojure Riemann configuration is Clojure code Built on Netty and Protocol Buffers Transient system Supports mutual TLS
  5. 5. Riemann Core Concepts
  6. 6. Riemann Core Concepts Riemann Event Riemann Index Riemann Stream
  7. 7. Riemann Event
  8. 8. Riemann Event
  9. 9. Riemann Index
  10. 10. Riemann Index Lock-free implementation of Java concurrent hashmap In memory Stores the latest event for a pair of (host, service) Event stays in index upto the ttl Riemann dash reads the index and displays the data
  11. 11. Riemann Stream
  12. 12. Riemann Stream Clojure Functions Filter events from stream Split a stream into multiple streams Merge multiple streams into one stream
  13. 13. Riemann Stream: Print
  14. 14. Riemann Stream: Filter
  15. 15. Riemann Stream: Filter
  16. 16. Riemann Stream: Split
  17. 17. Riemann Stream: Merge
  18. 18. Sending Data to Riemann
  19. 19. Sending Data to Riemann
  20. 20. Sending Data to Riemann Collectd Logstash Syslog-ng
  21. 21. Riemann Configuration
  22. 22. Riemann Configuration Riemann config is Clojure code Divide config into smaller, testable & reusable pieces using Clojure functions Organise them into Clojure namespaces
  23. 23. Configuration
  24. 24. Riemann Configuration: Organise /etc/riemann/myorg/infra.clj
  25. 25. Riemann Configuration: Organise /etc/riemann/riemann.config
  26. 26. Riemann Configuration: Testing Clojure tests Supports adding tests with riemann.test/tap Tap is like breakpoints where you would like to observe the stream
  27. 27. Riemann Configuration: Testing
  28. 28. Monitoring Examples
  29. 29. Detect Service Going Down TTL means how many seconds the event is valid for after its occurrence Once expired: - event is removed from index - state is changed to “expired” - reinjected into the stream
  30. 30. Things breaking Flood of events leading to flood of alerts Need to know that problem exists but don’t need to know every single failure
  31. 31. Things breaking: Rollup
  32. 32. Things breaking: Throttle
  33. 33. Things breaking: Rollup & Throttle
  34. 34. Scaling Riemann
  35. 35. Scaling Riemann No automatic sharding
  36. 36. Scaling Riemann We can make use of two streams: batch async-queue
  37. 37. Scaling Riemann
  38. 38. Thank you Resources: https://riemann.io/ https://github.com/riemann/riemann

×