Log management can be overwhelming! This talk provide an overview of log management. You will learn about different kinds of logs, structured/unstructured logs, retention, archival, and more. We will also briefly introduce several common tools in the log management space.

1. Log Management
An Introduction
Lennart Koopmann
Jordan Sissel

2. What is a Log?

3. time + data
What is a Log?

4. 37.5.55.31 - - [08/Apr/2014:15:31:30 -0400]
"GET /images/web/2009/banner.png HTTP/1.1"
304 - "http://semicomplete.com/style2.css"
"Mozilla/5.0 (Macintosh; Intel Mac OS X
10.9; rv:28.0) Gecko/20100101 Firefox/28.0"
What is a Log? time + data

5. 120707 0:40:34 4 Connect root@localhost on
4 Query select @@version_comment limit 1
120707 0:40:45 4 Query select * from mysql.user
What is a Log? time + data
?????? ???????

6. Kinds of Logs

7. Trace and Debug
Kinds of Logs

8. Accounting
Kinds of Logs

9. Transaction
Kinds of Logs

10. Problems

11. Difficult to Access
Problems

12. Too Many Logs
Problems Difficult to Access

13. Too Many Servers
Problems Difficult to Access

14. No Permissions :(
Problems Difficult to Access

15. Difficult to Consume
Problems

16. Unstructured
Problems Difficult to Consume

17. Requires Expertise
Problems Difficult to Consume

18. Requires Maintenance
Problems

19. Configuration
Problems Requires Maintenance

20. Log Retention
Problems Requires Maintenance

21. Bad Tooling
Problems

22. grep, ssh, awk
Problems Bad Tooling

23. Life of a Log

24. Life of a Log
Record
Transport
Search &
Analyze
Archive
Delete

25. Sources of Logs

26. Vendor Hardware
Sources of Logs
Routers, VPNs, Printers, Phones,
AWS CloudTrail, etc

27. Vendor Software
Sources of Logs
Nginx, Wordpress, Jira

28. In-house Software
Sources of Logs
Your company controls it

29. Solutions
(Open Source!)

30. Solutions
Logstash
from Elasticsearch
Graylog2
from Torch