This document discusses how to collect big data into Hadoop using Apache Flume and Fluentd. It describes some problems with a poor man's approach to data collection and discusses the basic theories of divide and conquer and streaming to make data collection more efficient. It then provides an overview of how Apache Flume and Fluentd work, including their network topologies, configurations, and plugin systems. Examples are given of how Fluentd has been used at Treasure Data to collect and analyze REST API logs, backend logs, and Hadoop logs. The document concludes with a discussion of developing plugins for Fluentd.

1. How to collect Big Data
into Hadoop
Big Data processing to collect Big Data
fluentd.org
Sadayuki Furuhashi

2. Self-introduction
> Sadayuki Furuhashi
> Treasure Data, Inc.
Founder & Software Architect
> Open source projects
MessagePack - efficient serializer (original author)
Fluentd - event collector (original author)

3. We’re Hiring!
sf@treasure-data.com

4. Today’s topic

5. Big Data
Report &
Monitor

6. Big Data
Collect Store Process Visualize
Report &
Monitor

7. easier & shorter time
Collect Store Process Visualize
Cloudera Excel
Horton Works Tableau
MapR R

8. How to shorten here? easier & shorter time
Collect Store Process Visualize
Cloudera Excel
Horton Works Tableau
MapR R

9. Problems to collect data

10. Poor man’s data collection
1. Copy files from servers using rsync
2. Create a RegExp to parse the files
3. Parse the files and generate a 10GB CSV file
4. Put it into HDFS

11. Problems to collect “big data”
> Includes broken values
> needs error handling & retrying
> Time-series data are changing and uncler
> parse logs before storing
> Takes time to read/write
> tools have to be optimized and parallelized
> Takes time for trial & error
> Causes network traffic spikes

12. Problem of poor man’s data collection
> Wastes time to implement error handling
> Wastes time to maintain a parser
> Wastes time to debug the tool
> Not reliable
> Not efficient

13. Basic theories
to collect big data

14. Divide & Conquer
error
error

15. Divide & Conquer & Retry
error retry
retry
error retry retry

16. Streaming
Don’t handle big files here Do it here

17. Apache Flume and Fluentd

18. Apache Flume

19. Apache Flume
access logs Agent
app logs Agent Collector
system logs Agent Collector
Agent
...

20. Apache Flume - network topology
Master
Agent ack
Agent Collector
Flume OG Collector
Agent Collector send
Agent
Agent
send/ack
Agent Collector
Flume NG Collector
Agent Collector
Agent

21. Apache Flume - pipeline
Source Sink
Flume OG
plugin
Source Channel Sink
Flume NG

22. Apache Flume - configuration
Master
Master manages
all configuration
(optional)
Agent
Agent Collector
Flume NG Collector
Agent Collector
Agent

23. Apache Flume - configuration
# source
host1.sources = avro-source1
host1.sources.avro-source1.type = avro
host1.sources.avro-source1.bind = 0.0.0.0
host1.sources.avro-source1.port = 41414
host1.sources.avro-source1.channels = ch1
# channel
host1.channels = ch_avro_log
host1.channels.ch_avro_log.type = memory
# sink
host1.sinks = log-sink1
host1.sinks.log-sink1.type = logger
host1.sinks.log-sink1.channel = ch1

24. Fluentd

25. Fluentd - network topology
Agent
send/ack
Agent Collector
Flume NG Collector
Agent Collector
Agent
fluentd
send/ack
fluentd fluentd
Fluentd fluentd
fluentd fluentd
fluentd

26. Fluentd - pipeline
Source Channel Sink
Flume NG
plugin
Input Buffer Output
Fluentd

27. Fluentd - configuration
fluentd
fluentd fluentd
Fluentd fluentd
fluentd fluentd
fluentd
Use chef, puppet, etc. for configuration
(they do things better)
No central node - keep things simple

28. Fluentd - configuration
<source>
type forward
port 24224
</source>
<match **>
type file
path /var/log/logs
</match>

29. Fluentd - configuration
# source
host1.sources = avro-source1
host1.sources.avro-source1.type = avro
<source> host1.sources.avro-source1.bind = 0.0.0.0
type forward host1.sources.avro-source1.port = 41414
port 24224 host1.sources.avro-source1.channels = ch1
</source>
# channel
<match **> host1.channels = ch_avro_log
type file host1.channels.ch_avro_log.type = memory
path /var/log/logs
</match> # sink
host1.sinks = log-sink1
host1.sinks.log-sink1.type = logger
host1.sinks.log-sink1.channel = ch1

30. Fluentd - Users

31. Fluentd - plugin distribution platform
$ fluent-gem search -rd fluent-plugin
$ fluent-gem install fluent-plugin-mongo

32. Fluentd - plugin distribution platform
$ fluent-gem search -rd fluent-plugin
$ fluent-gem install fluent-plugin-mongo
94 plugins!

33. Concept of Fluentd
Customization is essential
> small core + many plugins
Fluentd core helps to implement plugins
> common features are already implemented

34. Fluentd core Plugins
Divide & Conquer
Retrying
read / receive data
Parallelize
write / send data
Error handling
Message routing

35. Fluentd plugins

36. in_tail
apache
fluentd
access.log
✓ read a log file
✓ custom regexp
✓ custom parser in Ruby

37. out_mongo
apache
fluentd
in_tail
access.log buffer

38. out_mongo
apache
fluentd
in_tail
access.log buffer
✓ retry automatically
✓ exponential retry wait
✓ persistent on a file

39. out_s3
apache
fluentd
in_tail
access.log buffer Amazon S3
✓ slice files based on time
✓ retry automatically
2013-01-01/01/access.log.gz ✓ exponential retry wait
2013-01-01/02/access.log.gz ✓ persistent on a file
2013-01-01/03/access.log.gz
...

40. out_hdfs ✓ custom text formater
apache
fluentd
in_tail
access.log buffer HDFS
✓ slice files based on time
✓ retry automatically
2013-01-01/01/access.log.gz ✓ exponential retry wait
2013-01-01/02/access.log.gz ✓ persistent on a file
2013-01-01/03/access.log.gz
...

41. out_hdfs ✓ automatic fail-over
✓ load balancing
fluentd
apache
fluentd fluentd
in_tail
fluentd
access.log buffer
✓ slice files based on time
✓ retry automatically
2013-01-01/01/access.log.gz ✓ exponential retry wait
2013-01-01/02/access.log.gz ✓ persistent on a file
2013-01-01/03/access.log.gz
...

42. Fluentd examples

43. Fluentd at Treasure Data - REST API logs
fluent-logger-ruby
API servers + in_forward
Rails app
fluentd
Rails app
fluentd
out_forward
watch server fluentd

44. Fluentd at Treasure Data - backend logs
fluent-logger-ruby
API servers + in_forward worker servers
Rails app Ruby app
fluentd fluentd
Rails app Ruby app
fluentd fluentd
out_forward
watch server fluentd

45. Fluentd at Treasure Data - monitoring
fluent-logger-ruby
API servers + in_forward worker servers
Rails app Queue Ruby app
fluentd fluentd
PerfectQueue
Rails app Ruby app
fluentd fluentd
out_forward script
in_exec
fluentd watch server

46. Fluentd at Treasure Data - Hadoop logs
✓ resource consumption
statistics for each user Hadoop
✓ capacity monitoring JobTracker
thrift API call
script
in_exec
fluentd watch server

47. Fluentd at Treasure Data - store & analyze
fluentd watch server
out_tdlog out_metricsense
✓ streaming aggregation
Treasure Data Librato Metrics
for historical analysis for realtime analysis

50. Plugin development

51. class SomeInput < Fluent::Input
Fluent::Plugin.register_input('myin', self)
config_param :tag, :string
def start
Thread.new {
while true
time = Engine.new
record = {“user”=>1, “size”=>1}
Engine.emit(@tag, time, record)
end
}
end
def shutdown
...
end
end
<source>
type myin
tag myapp.api.heartbeat
</source>

52. class SomeOutput < Fluent::BufferedOutput
Fluent::Plugin.register_output('myout', self)
config_param :myparam, :string
def format(tag, time, record)
[tag, time, record].to_json + "n"
end
def write(chunk)
puts chunk.read
end
end
<match **>
type myout
myparam foobar
</match>

53. class MyTailInput < Fluent::TailInput
Fluent::Plugin.register_input('mytail', self)
def configure_parser(conf)
...
end
def parse_line(line)
array = line.split(“t”)
time = Engine.now
record = {“user”=>array[0], “item”=>array[1]}
return time, record
end
end
<source>
type mytail
</source>

55. Fluentd v11
Error stream
Streaming processing
Better DSL
Multiprocess