Sadayuki Furuhashi is the founder and software architect of Treasure Data, Inc. He is the original author of Fluentd and MessagePack. Treasure Data uses Fluentd to collect system metrics from applications, Hadoop, and databases and send them to services like Librato Metrics and Treasure Data for analysis and alerts in PagerDuty. The next version of Fluentd will include filter plugins and allow filtering pipelines within the source configuration.

1. Sadayuki Furuhashi
Treasuare Data, Inc.
Founder & Software Architect
How we use Fluentd
in Treasure Data
for system metrics
A creator of Fluentd

2. Self-introduction
> Sadayuki Furuhashi
> twitter/github: frsyuki
> Treasure Data, Inc.
Founder & Software Architect
> Open source projects
MessagePack - efficient serializer (original author)
Fluentd - event collector (original author)

5. Metrics collection from Rails apps
fluentd
Rails app
fluentd
fluentd
Rails app
MetricSense for Ruby
aggregate logs using
out_forward plugin
local fluentd collects logs
and buffer them in local disk

6. class ApplicationController
before_filter :setup_metrics
after_filter :post_metrics
def setup_metrics
MetricSense.clear!
MetricSense.tag "#{params[:controller]}_#{params[:action]}"
if current_user.present?
MetricSense.segment(
:account=>current_user.account_id,
:agent=>request.env[‘HTTP_AGENT’])
end
end
def post_metrics
MetricSense.measure!
end
# in application code:
# MetricSense.value :size=>10

7. Metrics collection from Hadoop
fluentd
Hadoop
Cluster
Hadoop
Clusterhadoop metrics plugin
(original plugin)
✓ number of failed tasks
✓ number of used slots
✓ blacklisted nodes
✓ memory usage of
JobTracker
✓ memory usage of
NameNode/DataNode
✓ HDFS put/get latency
etc...

8. Metrics collection from DB
fluentd
script in_exec plugin
runs a script every
1 hour
runs SQL statements
✓ users’ activity and stats
✓ job queue length
(PerfectQueue stats)
etc...

9. Metrics collection from Hadoop
fluentd
Rails app
fluentd
Rails app
fluentd
Hadoop
JobTracker
Hadoop
JobTracker
script

10. Metrics utilization
fluentd
Librato Metrics
for realtime analysis
Treasure Data
for historical analysis
Pager Duty
for alerts
out_tdlogout_metricsense

12. Next version of Fluentd
> <filter>
> <match> in <source>
> <label> for error records
> New process model & Live restart

13. <source>
type tail
path /var/log/httpd.log
format apache
tag not_filtered.apache
</source>
<match not_filetered.**>
type rewrite
remove_prefix not_filtered
<rule>
key status
pattern ^500$
ignore true
</rule>
</match>
<match **>
type forward
host log.server
</match>
Before
Mysterious tag
tag operations

14. <source>
type tail
path /var/log/httpd.log
format apache
tag apache
</source>
<filter **>
type rewrite
<rule>
key status
pattern ^500$
ignore true
</rule>
</filter>
<match **>
type forward
host log.server
</match>
After (v11)
Filter plugins!

15. <source>
type tail
path /var/log/httpd.log
format apache
tag apache
<filter **>
type rewrite
<rule>
key status
pattern ^500$
ignore true
</rule>
</filter>
</source>
<match **>
type forward
host log.server
</match>
After (v11)
<filter>/<match> in
<source>

16. <source>
type tail
path /var/log/httpd.log
tag apache
</source>
<match **>
type forward
host log.server
</match>
Before
I want to add flowcounter here...

17. <source>
type tail
path /var/log/httpd.log
tag apache
</source>
<match flow.traffic>
type forward
host traffic.server
</match>
<match **>
type copy
<store>
type flowcounter
tag flow.traffic
</store>
<store>
type forward
host log.server
</store>
</match>
Before
Nested!

18. <source>
type tail
path /var/log/httpd.log
tag apache
</source>
<filter **>
type copy
<match>
type flowcounter
tag flow.traffic
<match>
type forward
host traffic.server
</match>
</match>
</filter>
<match **>
type forward
host log.server
</match>
After (v11)
Filtering pipeline

19. <source>
type forward
</source>
<filter **>
type copy
<match>
type file
path /mnt/local_archive
</match>
</filter>
<label alert>
<match **>
...
</match>
</label>
<label analysis>
...
</label>
# copy & label & forward
<filter **>
type copy
<match>
type forward
label alert
host alerting.server
</match>
</filter>
# copy & label & forward
<filter **>
type copy
<match>
type forward
label analysis
host analysis.server
</match>
</filter>
After (v11)