Sematext's DevOps Evangelist, Stefan Thies (@seti321), takes a Docker Logging tour through the different log collection options Docker users have, the pros and cons of each, specific and existing Docker logging solutions, tooling, the role of syslog, log shipping to ELK Stack, and more. Q&A session at end.

1. Docker Logging
Webinar
20% OFF 201509WNR20S 201509WNR20L
sematext.com/spm sematext.com/logsene

2. Housekeeping / Questions

3. Intro
Logsene: Centralized Log Management
Search and Big Data Consulting
Support for Solr and Elasticsearch
SPM: Performance monitoring,
Anomaly Detection and Alerting

4. SPM - Performance Monitoring

5. Logsene - Log Management

6. Agenda
● Centralized Log Management
● Docker - What is different?
○ Challenges
○ How to
■ Log Drivers
■ Logging Containers
■ Sematext Solutions

7. Centralized Log Management
error: No space
left on device
/dev/...
?
warn: Transaction
“order_product”
failed!

8. a few steps to go ...
Log Shippers Centralized Log Management / Logsene
Server,
Container,
Application
Use
JSON,
Luke

9. Structured Data

10. Docker Logging Challenges
● Access Logs
● Log Forwarding to central data stores
● Log Parsing
● Deployment of Logging Tools
○ Containers on local Host
○ Separate Hosts
○ SaaS

11. What are Docker Logs?
● Traditionally separate files for
each Application and Log-Type
○ error.log
○ access.log
● Docker Logs are stdout / stderr of
processes running in a container
● Most official images log to console

12. Mixed Log Formats in one Container

13. Docker Logging Options
- Docker Log Drivers
- json-file, syslog, fluentd,
journald, gelf
- Docker API based Logging
Containers
- Logspout
- Sematext Docker Container
- Custom images with installed
log shipper (syslog)

14. Docker Log Drivers
Cons:
- No Log Parser - only Log Forwarding
- “docker logs” command works only
with Log-Driver “JSON-files”
- Containers terminate when the TCP
Server (e.g. syslog or fluentd) is not
reachable
- No TLS encryption for syslog
Pros:
- Simple way to forward logs to remote
destinations
- Setup per container or global setting
for Docker

15. Example: Log Drivers
# Start a syslog server :)
logagent -u 1514 -y -t af648d4f-xxxx-xxxx-8ec0-fcb33f884f57
# Start a Web Server with TCP syslog -> container terminates
docker run -d --name my_web_app -p 80:80 --log-driver=syslog --log-opt syslog-
address=tcp://localhost:1514 httpd
# Start a Web Server with UDP syslog -> container starts
docker run -d --name my_web_app -p 80:80 --log-driver=syslog --log-opt syslog-
address=udp://localhost:1514 httpd
# run docker logs -> fails
docker logs my_web_app
> logsene search http

16. Logging Containers: Logspout
Pros:
- Logging does not affect app
container
- ANSI Escape Sequence removal
- TLS support
- Real-time View with HTTP API
- Config for Filters and Syslog-Tags
- Log-Driver Files / journald Logs
are available on the Host
Cons:
- Logging Container must be online
- Only forwarding, no Log Parser,
rsyslog could be used for parsing
- Limited to log collection

17. Logspout HTTP View

18. Logging Containers: SPM for Docker
Pros:
- ANSI Escape Sequence handling
- TLS by default
- Near Real-time View in UI
- Filters by regex for Image,
Container Names
- Structured Logs with included
Log-Parser and Pattern Library
- Collects Logs, Metrics and
Events
- Hosted ELK Stack in Logsene
Cons:
- Logging container must be online

19. Demo
docker run -d --name sematext-agent
-v /var/run/docker.sock:/var/run/docker.sock
-v $PWD/patterns.yml:/etc/logagent/patterns.yml
-e HOSTNAME=$HOSTNAME
-e LOGSENE_TOKEN=53a6c7e7-xxxx-4725-962e-ea47cebxxx
-e SPM_TOKEN=fe31fc3a-xxxx-47c6-b83c-be376bfxxx
sematext/spm-agent-docker
docker run --name webapp -p 80:80 httpd
siege localhost:80/unknow_page.html
logsene search error

20. Logs
Logsene
Token
Metrics +
Events
Docker logs on CoreOS
Web UI
Sematext Container
Logsene
(https)
SPM
(https)
Log forwarding
service
stores status in etcd
Logging
Gateway
(TCP 9000)
Docker Daemon
API / unix-socket
Events
Metrics
Logs
etcd
journald
Configuration in etcd
- Logsene Token
- SPM Token
Logging gateway port,
Logging status per host
Journald
Logs
SPM
Token

21. Containerized Monitoring & Logging
SPM Performance Monitoring and Logsene
Metrics, Events and Logs

22. SPM Logsene
METRICS + LOGS ⇒ BETTER TOGETHER

23. Mixed Log Formats in one Container

24. Parsed Logs from a mixed stream

25. Making Logs Analytics-ready
Log
Parser
Inside
Reduced Stack for
Logging!
Structured Data
for Analytics

26. Summary
Stefan Thies
Twitter: @seti321
stefan.thies@sematext.com
info@sematext.com
blog.sematext.com
sematext.com/logsene
hub.docker.com/r/sematext/spm-agent-docker/
github.com/sematext/spm-agent-docker

27. Docker Logging
Webinar
20% OFF 201509WNR20S 201509WNR20L
sematext.com/spm sematext.com/logsene

28. Thank you for your attention