10. Fluentd & Event logs
Before:
App server App server App server
Application Application Application
File File File ... File File File ... File File File ...
File
High latency
must wait for a day
Log server Hard to analyze
complex text parsers
11. Fluentd & Event logs
After:
App server App server App server
Application Application Application
Fluentd Fluentd Fluentd
Realtime!
Fluentd Fluentd
13. # receive events via HTTP # save alerts to a file
<source> <match alert.**>
type http type file
port 8888 path /var/log/fluent/alerts
</source> </match>
# read logs from a file # forward other logs to servers
<source> # (load-balancing + fail-over)
type tail <match **>
path /var/log/httpd.log type forward
format apache <server>
tag apache.access host 192.168.0.11
</source> weight 20
</server>
# save access logs to MongoDB <server>
<match apache.access> host 192.168.0.12
type mongo weight 60
host 127.0.0.1 </server>
</match> </match>
14. Fluentd vs Scribe
• Deals with structured logs
• Easy to install
> “gem install fluentd”
> apt-get and yum
http://packages.treasure-data.com/
• Easy to customize
• add/modify plugins without re-compiling
> “gem search -rd fluent-plugin”
15. Fluentd vs Flume
• Easy to setup
> “sudo fluentd --setup && fluentd”
• Very small footprint
> small engine (3,000 lines) + plugins
• JVM-free
• Easy to configure
29. Plugins
• Bundled plugins
> file
writes event logs to files hourly or daily
> forward
forwards event logs (+fail-over and load balancing)
> exec
passes event logs to/from external commands
> tail
reads event logs from a file (like `tail -f`)
30. Plugins
• 3rd party plugins
> scribe
integrates Fluentd with Scribe
> s3
uploads log files to Amazon S3 hourly or daily
> mongo
writes logs to MongoDB
> hoop
puts log files on Hadoop HDFS via Hoop
...
31. Plugin developer API
• Unit test framework (like “MRUnit”)
> Fluent::Test::InputTestDriver
> Fluent::Test::OutputTestDriver
> Fluent::Test::BufferedOutputTestDriver
• Fluent::TailInput (base class of “tail” plugin)
> text parser is customizable
def parse_line(line)