Fluentd and docker monitoring @ dockerbangalore meetup

1. FluentD and Docker
Vinay Krishna
SolutionsIQ

2. Monitoring

3. Is it simple?
• Monitoring
• Applications
• Servers

5. Logging

6. What?
• Status of application
• Keeps information about errors/failure
• Status of Network

7. Why?
• Developers
• Get help in Debugging
• IT admin / support
• Get help in Trouble-shooting
• Apps running smoothly
• Security
• Business
• Input data – analytics
• User interaction / behaviors
• Improvements

8. Assumptions
• I have enough disk space
• I/O operations will not block
• Log messages are human readable
• My logging mechanism scale
• Basically, yeah.. it should work.

9. Concerns
• Logs increase = data increase
• Message format get more complex
• Did the Kernel flush the buffers ? (sync(2))
• Multi-thread application ?, locking ?
• Multiple Applications = Multiple Logs
• If Multiple Applications = Multiple logs
• Multiple Hosts x Multiple Applications = ???

10. How to parse/store multiple data sources ?

11. Fluentd is an open source data collector for unified logging
layer.
It allows you to unify data collection and consumption for a
better use and understanding of data.
• Structured logging
• Reliable forwarding
• Pluggable architecture

12. Fluentd
• Data collection for unified logging layer
• Streaming data transfer based on JSON
• Written in Ruby
• Gem based various plugins
• http://www.fluentd.org/plugins
• Working on lots of productions
• http://www/fluentd.org/testimonials

13. Before

14. After

16. Highlights
• Unified Logging Layer
• Fluentd tries to structure data as JSON as much as possible
• Simple and yet flexible
• 300+ plugins
• Open Source
• Proven Reliability and Performance
• 2000+ data-driven companies rely on FluentD
• Minimum resources required - vanilla instance runs on 30-40MB of
memory and can process 13,000 events/second/core
• Data loss should never happen.
• Fluentd supports memory- and file-based buffering to prevent inter-node
data loss.
• Fluentd also supports robust failover and can be set up for high availability
• Community

18. Docker Monitoring

19. Monitor
• Resource utilization
• How much RAM and CPU is each container using?
• Health of docker environments
• As the Docker ecosystem continues to evolve, we have to ask ourselves
the following questions:
• How can we log and monitor Docker effectively?
• This includes logging the Docker runtime infrastructure, the container itself and
what goes on inside of it, and how to ensure to collect log data from ephemeral
containers.
• How can we use feedback from containers to manage and improve the
quality of our services?
• Can we build off of decades of experience logging monolithic applications,
or do we have to start from scratch?
• If we have to start from scratch, how can we build a solution that helps us
make better decisions?

20. FluentD + Docker

21. Logging of container architecture
• Storage:
• should be outside of container / hosts
• Transferring:
• should be over network
• Aggregation:
• should be done per container / per service

22. FluentD Architecture

30. Logging Driver
• Docker v1.6 released the concept of logging drivers
• Route container output
• Add new logging driver – fluentd
• --log-driver=fluentd
• https://github.com/docker/docker/pull/12876
• New for docker v1.7.0?

31. Container logging driver “fluentd”
• Apps write logs to STDOUT:
• docker sends it to fluentd
directly!
• Pros:
• simple conf for apps and
docker
• logs include container logs
• Cons:
• ?

32. Fluentd docker image
• Official image by fluentd organization
https://registry.hub.docker.com/u/fluent/fluentd/
• Use it as it is, or build your own container!
https://github.com/fluent/fluentd-docker-image

33. Demo

34. Install fluentd
• Install fluentd via td-agent
curl -L http://toolbelt.treasuredata.com/sh/install-
ubuntu-trusty-td-agent2.sh | sh
• Start td-agent
sudo /etc/init.d/td-agent start

35. Verify installation
• Check the logs to make sure it was installed
successfully
• tail /var/log/td-agent/td-agent.log

36. Build fluentd image
• Create a new directory for your Fluentd Docker resources, and move into it
• mkdir ~/fluentd-docker && cd ~/fluentd-docker
• Create the following Dockerfile
• sudo nano Dockerfile
• Add the following content:
FROM ruby:2.2.0
MAINTAINER kiyoto@treausuredata.com
RUN apt-get update
RUN gem install fluentd -v "~>0.12.3"
RUN mkdir /etc/fluent
RUN apt-get install -y libcurl4-gnutls-dev make
RUN /usr/local/bin/gem install fluent-plugin-elasticsearch
ADD fluent.conf /etc/fluent/
ENTRYPOINT ["/usr/local/bundle/bin/fluentd", "-c", "/etc/fluent/fluent.conf"]

37. Build fluentd image
• Create a fluent.conf file in the same directory
• sudo nano fluent.conf

38. <source>
type tail
read_from_head true
path /var/lib/docker/containers/*/*-json.log
pos_file /var/log/fluentd-docker.pos
time_format %Y-%m-%dT%H:%M:%S
tag docker.*
format json
</source>
# Using filter to add container IDs to each event
<filter docker.var.lib.docker.containers.*.*.log>
type record_transformer
<record>
container_id ${tag_parts[5]}
</record>
</filter>
<match docker.var.lib.docker.containers.*.*.log>
type elasticsearch
logstash_format true
host "#{ENV['ES_PORT_9200_TCP_ADDR']}" # dynamically configured to use Docker's link feature
port 9200
flush_interval 5s
</match>

39. • Build docker image
• docker build -t fluentd-es .
• Check successfully built the images
• docker images

40. ElasticSearch Container
• Move to home directory
• Cd ~
• Download and start the Elasticsearch container
• docker run -d -p 9200:9200 -p 9300:9300 --name es
elasticsearch
• Check elasticsearch container is running
• docker ps

41. Start the Fluentd-to-Elasticsearch
Container
• Start the container that runs Fluentd, collects the
logs, and sends them to Elastcisearch
• docker run -d --link es:es -v
/var/lib/docker/containers:/var/lib/docker/containers
fluentd-es
• Check that container is running
• docker ps

42. Confirm ElasticSearch receives
events
• curl -XGET 'http://localhost:9200/_all/_search?q=*‘

43. What’s next
• Setup Dashboard

44. References
• http://www.fluentd.org/guides/recipes/docker-
logging
• http://www.slideshare.net/repeatedly/docker-and-
fluentd-51821582
• http://www.slideshare.net/tagomoris/docker-and-
fluentd-revised
• https://www.socallinuxexpo.org/sites/default/files/
presentations/fluentd.pdf

45. Questions?