O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Fluentd Project Intro at Kubecon 2019 EU

758 visualizações

Publicada em

https://events.linuxfoundation.org/events/kubecon-cloudnativecon-europe-2019/schedule/

Publicada em: Tecnologia
  • Seja o primeiro a comentar

Fluentd Project Intro at Kubecon 2019 EU

  1. 1. Fluentd Project Intro Masahiro Nakagawa
 Senior Software Engineer    CNCon / KubeCon EU Barcelona 2019
  2. 2. Fluentd Overview
  3. 3. • Streaming data collector for unified logging • Core + Plugins • RubyGems based various plugins • Follow Ruby’s standard way • Several setup ways • https://docs.fluentd.org/installation • Latest version: v1.5.0 • Logging part in CNCF and Graduated at 2019 What’s Fluentd
  4. 4. Streaming way with Fluentd Log Server Application Server A File FileFile Application Server C File FileFile Application Server B File FileFile Low latency! Seconds or minutes Easy to analyze!! Parsed and formatted
  5. 5. LOG Unified logging layer M + N
  6. 6. Fluentd Architecture
  7. 7. • Buffering & Retrying • Error handling • Event routing • Parallelism
 • Helper for plugins Design Core Plugins • Read / receive data • Parse data • Filter / enrich data • Buffer data • Format data • Write / send data
  8. 8. • Nano-second unit • from logs Event structure Time Record • JSON object,
 not raw string Tag • for event routing • Identify data source { “str_field”:”hey”, “num_field”: 100, “bool_field”: true, “array_field”: [“elem1”, “elem2”]
 }
  9. 9. Data pipeline (simplified) Plugin Input Filter Buffer Output Plugin Plugin Plugin 2018-05-01 15:15:15 myapp.buy Time Tag Record { “user”:”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing”
 }
  10. 10. Architecture: Input Plugins HTTP+JSON (in_http) Local files (in_tail) Syslog (in_syslog) … Receive or pull logs from data sources Emit logs to data pipeline
 Parse incoming logs for
 structured logging Plugin Input
  11. 11. Filter Architecture: Filter Plugins Transform logs Filter out unnecessary logs Enrich logs Plugin Modify logs (record_transformer) Filter out logs (grep) Parse field (parser) …
  12. 12. Buffer Architecture: Buffer Plugins Plugin Improve performance Provide reliability Provide thread-safety Memory (buf_memory) File (buf_file)
  13. 13. Buffer Architecture: Buffer Plugins Chunk Plugin Input Output Chunk Chunk Improve performance Provide reliability Provide thread-safety
  14. 14. Architecture: Output Plugins Output Write or send event logs Plugin Local File (out_file) Amazon S3 (out_s3) Forward to other fluentd (out_forward) … Sync or Async
  15. 15. Retry Error Retry Batch Stream Error Retry Retry Divide & Conquer for retry Secondary
  16. 16. 3rd party input plugins dstat df AMQL munin SQL
  17. 17. 3rd party output plugins Graphite
  18. 18. Use-cases with Configuration Example
  19. 19. Simple forwarding
  20. 20. # logs from a file <source> @type tail path /var/log/httpd.log pos_file /tmp/pos_file <parse> @type apache2 </parse> tag app.apache </source> # logs from client libraries <source> @type forward port 24224 </source> # store logs to MongoDB <match app.**> @type mongo database fluent collection logs <buffer tag> @type file path /tmp/fluentd/buffer flush_interval 30s </buffer> </match>
  21. 21. All data Multiple destinations Hot data
  22. 22. # logs from a file <source> @type tail path /var/log/httpd.log pos_file /tmp/pos_file <parse> @type apache2 </parse> tag app.access </source> # logs from client libraries <source> @type forward port 24224 </source> # store logs to ES and HDFS <match app.**> @type copy <store> @type elasticsearch logstash_format true </store> <store> @type webhdfs host namenode port 50070 path /path/on/hdfs/ </store> </match>
  23. 23. Multi-tier Forwarding - At-most-once / At-least-once
 - HA (failover) - Load-balancing - keepalive forwarders aggregators https://www.slideshare.net/repeatedly/fluentd-and-distributed-logging-at-kubecon
  24. 24. Container and Kubernetes
  25. 25. Container Logging
  26. 26. • Docker : fluentd-docker-image • Alpine / Debian images • x86, Arm, PowerPC, etc support by Docker official • Kubernetes : fluentd-kubernetes-daemonset • Debian images • Some built-in destinations, ES, kafka, graylog, etc… • Helm chart • https://github.com/helm/charts/tree/master/stable/fluentd Resources
  27. 27. Docker logging with --log-driver=fluentd Server Container App FluentdSTDOUT / STDERR docker run --log-driver=fluentd 
 --log-opt fluentd-address=localhost:24224 { “container_id”: “ad6d5d32576a”, “container_name”: “myapp”, “source”: stdout } <source> @type forward </source>
  28. 28. Data collection with fluent-logger Server Container App Fluentd from fluent import sender from fluent import event sender.setup('app.events', host='localhost') event.Event('purchase', { 'user_id': 21, 'item_id': 321, 'value': '1' }) tag = app.events.purchase { “user_id”: 21, “item_id”: 321 “value”: 1, } fluent-logger library <source> @type forward </source>
  29. 29. Shared data volume and tailing Server Container App Fluentd <source> @type tail path /mnt/*/access.log pos_file /var/log/fluentd/access.log.pos <parse> @type nginx </parse> tag nginx.access </source> /mnt/nginx/logs
  30. 30. Kubernetes Daemonset Node Pod App Fluentd <source> @type tail path /var/log/containers/*.log pos_file /var/log/fluentd/access.log.pos <parse> @type json </parse> tag kubernetes.* </source> /var/log/containers
  31. 31. Kubernetes Daemonset & metadata Node Pod App Fluentd /var/log/containers <filter kubernetes.*> @type kubernetes_metadata_filter </filter> API { "log": "hellon", "stream": "stdout", "time": "2018-12-11T12:00:00.601357200Z" } { "log": "hellon", "kubernetes": { "namespace_name": "default", "container_name": "test-app-container", "namespace_labels": {
 "product_version": "v1.0.0" } … }
  32. 32. Container Logging approach summary • Collect log messages with docker • --log-driver=fluentd • Application data/metrics • fluent-logger • Access logs, logs from middleware • Shared data volume with in_tail • Kubernetes Daemonset • Collect container logs from /var/log/containers/* • Add kubernetes metadata to logs
  33. 33. Fluent-bit
  34. 34. Fluentd and Fluent-bit Fluentd Fluent-bit Implementation Ruby + C C Focus Flexibility and Robustness Performance and footprint Design Pluggable Pluggable Target Forwarder / Aggregator Forwarder / Edge Forward logs from fluent-bit to fluentd is popular pattern
  35. 35. Container Logging with fluent-bit
  36. 36. Enjoy logging

×