Fluentd is a data collection tool for unified logging that allows for extensible and reliable data collection. It uses a simple core with plugins to provide buffering, high availability, load balancing, and streaming data transfer based on JSON. Fluentd can collect log data from various sources and output to different destinations in a flexible way using its plugin architecture and configuration files. It is widely used in production for tasks like log aggregation, filtering, and forwarding.

1. FLUENTD: UNIFIED LOGGING
LAYER
John Hammink
October 14, 2015
Cask Big Data Application Meetup

2. About Me
• A recovering software & QA engineer turned digital
artist once interested in fractals;
• now into data visualization based on large datasets
rendered directly to GPU (RGL, various Python GL
libraries, etc.)
• github: jammink2; twitter: rijksband

3. Tweet NOW!
“At @caskdata learning how to collect
more event data using #Fluentd”

4. WHAT’S FLUENTD?
An extensible & reliable data collection
tool
simple core + plugins
buffering, HA (failover),
load balancing, etc.
like syslogd

5. What’s Fluentd?
> Data collector for unified logging layer
> Streaming data transfer based on JSON
> Written in Ruby
> Gem based various plugins
> http://www.fluentd.org/plugins
> Working in production
> http://www.fluentd.org/testimonials

6. data collection tool

7. ✓ duplicated code for error handling...
✓ messy code for retrying mechnism...
Blueflood
MongoDB
Hadoop
Metrics
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Your system
bash scripts ruby scripts
rsync
log file
bash
python scripts
custom
loggger
cron
other custom
scripts...

8. (this is painful!!!)

9. Blueflood
MongoDB
Hadoop
Metrics
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Your system
filter / buffer / route

10. extensible

11. CORE PLUGINS
• Divide & Conquer
• Buffering & Retries
• Error Handling
• Message Routing
• Parallelism
• Read Data
• Parse Data
• Buffer Data
• Write Data
• Format Data
Common
Concerns
Use Case
Specific

12. architecture

13. INTERNAL ARCHITECTURE
“input-ish” “output-ish”
Input Parser Buffer Output FormatterFilter

14. Internal Architecture (Simplified)
Input Buffer Output
Plugin Plugin Plugin
2012-02-04 01:33:51
myapp.buylog {
“user”: ”me”,
“path”: “/buyItem”,
“price”: 150,
“referer”: “/landing”
}
time
tag
record

15. Architecture :: Input plugins
Input
HTTP+JSON (in_http)
File tail (in_tail)
Syslog (in_syslog)
...
Plugin
✓ Receive logs
✓ Or pull logs from data sources
✓ in non-blocking manner

16. Architecture :: Output plugins
Plugin
✓ Write or send event logs
Output
File (out_file)
Amazon S3 (out_s3)
MongoDB (out_mongo)
...

17. Architecture :: Buffer plugins
Plugin
✓ Improve performance
✓ Provide reliability
✓ Provide thread-safety
Buffer
Memory (buf_memory)
File (buf_file)

18. Architecture :: Buffer plugins
Plugin
✓ Improve performance
✓ Provide reliability
✓ Provide thread-safety
chunk
chunk
chunk output
Input

20. reliable data transfer

21. DIVIDE & CONQUER & RETRY
error retry
error retry retry
retry

22. reliable process

23. THIS?

24. OR THIS?

25. M X N → M + N
Nagios
MongoDB
Hadoop
Alerting
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Databases
buffer/filter/route

26. use cases

27. SIMPLE FORWARDING

28. # logs from a file
<source>
type tail
path /var/log/httpd.log
format apache2
tag backend.apache
</source>
# logs from client libraries
<source>
type forward
port 24224
</source>
# store logs to MongoDB
<match backend.*>
type mongo
database fluent
collection test
</match>

29. LESS SIMPLE FORWARDING

30. LAMBDA ARCHITECTURE

31. # logs from a file
<source>
type tail
path /var/log/httpd.log
format apache2
tag web.access
</source>
# logs from client libraries
<source>
type forward
port 24224
</source>
# store logs to ES and HDFS
<match *.*>
type copy
<store>
type elasticsearch
logstash_format true
</store>
<store>
type webhdfs
host namenode
port 50070
path /path/on/hdfs/
</store>
</match>

32. FLUENTD ON KUBERNETES (NOV 2015)

33. FLUENTD LOGGING DRIVER (APR 2015)

34. Tweet Again!
“Happy v1 #k8s and congrats #Fluentd for
becoming a #docker logging driver”

35. DEMO: FLUENTD + DOCKER

36. THANK YOU!
AND TREASURE DATA IS HIRING!
WWW.TREASUREDATA.COMC/CAREERS