Internal training @ inuits

1. ;
Logstash and friendsLogstash and friends
Julien PivottoJulien Pivotto
Techies Teach TechiesTechies Teach Techies
September 2, 2013September 2, 2013

2. ;
Introduction
Logstash
Kibana
Conclusion
1 Introduction
2 Logstash
Missions
Inputs
Filters
Output
3 Kibana
4 Conclusion
Julien Pivotto Logstash and friends

3. ;
Introduction
Logstash
Kibana
Conclusion
Logging
• Recording of eventsRecording of events
• Voice of your systems and applicationsVoice of your systems and applications
• It tells you almost everythingIt tells you almost everything
• It is a source of knowledgeIt is a source of knowledge
Julien Pivotto Logstash and friends

4. ;
Introduction
Logstash
Kibana
Conclusion
Logging is useful
• Understanding outagesUnderstanding outages
Julien Pivotto Logstash and friends

5. ;
Introduction
Logstash
Kibana
Conclusion
Logging is useful
• Understanding outagesUnderstanding outages
• not only when it’s wrongnot only when it’s wrong
• you can extract metricsyou can extract metrics
• no logs means somethingno logs means something
• it tells you what, why, who, whenit tells you what, why, who, when
Julien Pivotto Logstash and friends

6. ;
Introduction
Logstash
Kibana
Conclusion
Logging in the wild
• SyslogSyslog
• |tee /var/log/myapp.log|tee /var/log/myapp.log
• Cron + MAILTO=Cron + MAILTO=
• &>/dev/null&>/dev/null
Julien Pivotto Logstash and friends

7. ;
Introduction
Logstash
Kibana
Conclusion
Logging in the past
• Logging to files on each serverLogging to files on each server
• Using syslog protocolUsing syslog protocol
• DecentralizedDecentralized
• Reading requires SSH accessReading requires SSH access
• Not developer friendlyNot developer friendly
Julien Pivotto Logstash and friends

8. ;
Introduction
Logstash
Kibana
Conclusion
The tools nowadays
• Jenkins, Icinga, Graphite, ForemanJenkins, Icinga, Graphite, Foreman
• Nice web interfacesNice web interfaces
• CentralizedCentralized
• Easy to useEasy to use
Julien Pivotto Logstash and friends

9. ;
Introduction
Logstash
Kibana
Conclusion
Requirements
• Scalable toolsScalable tools
• Configured by text filesConfigured by text files
• Playing with existing toolsPlaying with existing tools
• ScalableScalable
• Following the Unix philosophyFollowing the Unix philosophy
Julien Pivotto Logstash and friends

10. ;
Introduction
Logstash
Kibana
Conclusion
3 separate tools
• Elasticsearch, distributed search & analytics engineElasticsearch, distributed search & analytics engine
• Logstash, logs managementLogstash, logs management
• Kibana, very nice webui to ES and LogstashKibana, very nice webui to ES and Logstash
Julien Pivotto Logstash and friends

11. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Logstash
Julien Pivotto Logstash and friends

12. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Shipping the logs
• Some applications can only write to filesSome applications can only write to files
• But you need them on the main logstash serverBut you need them on the main logstash server
• Logstash can act as a daemon to ship the logsLogstash can act as a daemon to ship the logs
• Destinations can be syslog, redis,. . .Destinations can be syslog, redis,. . .
• Then you can act on your logsThen you can act on your logs
Julien Pivotto Logstash and friends

13. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Collecting the logs
• You can plug logstash to a lot of data sourcesYou can plug logstash to a lot of data sources
• It can be passive or activeIt can be passive or active
• Listening on a UDP port vs checking mailsListening on a UDP port vs checking mails
• All your logs are managed by one applicationAll your logs are managed by one application
• It creates fields from the logsIt creates fields from the logs
Julien Pivotto Logstash and friends

14. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Filtering the logs
• Making sense of a log messageMaking sense of a log message
• Finding what is importantFinding what is important
• Adding and removing fieldsAdding and removing fields
Julien Pivotto Logstash and friends

15. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Storing the logs
• Output to ElasticsearchOutput to Elasticsearch
• Sending information to statsdSending information to statsd
• Sending to your inbox, to icinga or filesSending to your inbox, to icinga or files
Julien Pivotto Logstash and friends

16. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
What is an "event"?
• Several fieldsSeveral fields
• Several tagsSeveral tags
• A type (syslog message, irc message,. . . )A type (syslog message, irc message,. . . )
• A @message fieldA @message field
• A timestampA timestamp
Julien Pivotto Logstash and friends

17. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Event
Julien Pivotto Logstash and friends

18. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
http://www.flickr.com/photos/quinnanya/7237788632/
Julien Pivotto Logstash and friends

19. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
UDP and TCP input
• Compatible with rsyslog protocolCompatible with rsyslog protocol
• Each syslog talks with logstash directlyEach syslog talks with logstash directly
• Allow you to use the syslog toolchains: logger, rsyslogAllow you to use the syslog toolchains: logger, rsyslog
• UDP is shoot and forgetUDP is shoot and forget
Julien Pivotto Logstash and friends

20. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
UDP and TCP input
Logstash configuration
input {
udp {
type => syslog
port => 5544
}
tcp {
type => syslog
port => 5544
}
}
Julien Pivotto Logstash and friends

21. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
UDP and TCP input
Rsyslog configuration
*.* @logstash.example.com:5544
• IIn /etc/rsyslog.conf
• TThat line will forward all the logs to logstash
• LLogstash will make useful fields out of it: priority, severity,
program. . .
Julien Pivotto Logstash and friends

22. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
File
• Enable you to use logstash with every applicationEnable you to use logstash with every application
• Useful to ship the logsUseful to ship the logs
• Acts as a tail -n 0 -FActs as a tail -n 0 -F
• It works even if you use logrotateIt works even if you use logrotate
Julien Pivotto Logstash and friends

23. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
File
input {
file {
path => "/var/log/legacyapp.log"
type => "legacylog"
}
}
Julien Pivotto Logstash and friends

24. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Grok
• Extract fields from textExtract fields from text
• Useful to read messagesUseful to read messages
• A lot of pre-existing patternsA lot of pre-existing patterns
• Uses Regex to find out fieldsUses Regex to find out fields
Julien Pivotto Logstash and friends

25. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Grok
Input text
Invalid user oracle from 85.249.144.18
Grok pattern
Invalid user %{USERNAME:login} from %{IP:ip}
Result
{
"login": [
[ "oracle" ]
],
"ip": [
[ "85.249.144.18" ]
]
}
Julien Pivotto Logstash and friends

26. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Grok
filter {
grok {
type => "syslog"
pattern => ["(?m)<%{POSINT:syslog_pri}>..."
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{@source_host}" ]
add_tag => "syslog-%{syslog_program}"
}
}
Julien Pivotto Logstash and friends

27. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Grep
• Allows you to grep interresting messagesAllows you to grep interresting messages
• Useful to countUseful to count
Julien Pivotto Logstash and friends

28. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Grep
filter {
grep {
add_field => ["outputirc", "A puppet package
has been deployed"]
add_tag => "outputirc"
drop => false
match => [ "syslog_program", "yum" ]
match => [ "@source_host", "puppetmaster" ]
match => [ "@message", "puppet-tree" ]
}
}
Julien Pivotto Logstash and friends

29. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Geoip
filter{
geoip {
tags => ["syslog-httpd"]
source => ["client"]
}
}
• Transform ip address into geo dataTransform ip address into geo data
• Useful to filter by country/map the dataUseful to filter by country/map the data
Julien Pivotto Logstash and friends

30. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Elasticsearch
• Version of elasticsearch <=> version of logstashVersion of elasticsearch <=> version of logstash
• Unless you use the elasticsearch_http outputUnless you use the elasticsearch_http output
output {
elasticsearch {
}
}
Julien Pivotto Logstash and friends

31. ;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
IRC
output {
irc {
channels => ["#example"]
host => "chat.freenode.net"
nick => "loggy"
port => 6667
tags => "outputirc"
user => "loggy"
format => "%{outputirc}"
}
}
Julien Pivotto Logstash and friends

32. ;
Introduction
Logstash
Kibana
Conclusion
statsd
output {
statsd {
host => ’127.0.0.1’
sender => "logstash"
increment => [ "httpd.%{http_host}.r.%{response}",
"httpd.response.%{response}"]
count => ["apache.%{http_host}.bytes", "%{bytes}" ]
timing => ["apache.%{http_host}", "%{duration_msec}"]
tags => ’grokked-apache’
}
}
Julien Pivotto Logstash and friends

33. ;
Introduction
Logstash
Kibana
Conclusion
Kibana
• Kibana is a web interface for Logstash/ESKibana is a web interface for Logstash/ES
• Kibana 1 was written in PHPKibana 1 was written in PHP
• Kibana 2 was written in RubyKibana 2 was written in Ruby
• Kibana 3 is written in AngularJSKibana 3 is written in AngularJS
Julien Pivotto Logstash and friends

34. ;
Introduction
Logstash
Kibana
Conclusion
Kibana 3
• Everything happens in the browserEverything happens in the browser
• The browser is connected to ElasticsearchThe browser is connected to Elasticsearch
• You can save dashboards into ESYou can save dashboards into ES
• You can write/template dashboards to filesYou can write/template dashboards to files
Julien Pivotto Logstash and friends

35. ;
Introduction
Logstash
Kibana
Conclusion
Installing kibana3
git clone https://github.com/elasticsearch/kibana.git
ssh -NL 9200:127.0.0.1:9200 elasticsearch &
python -m SimpleHTTPServer
Julien Pivotto Logstash and friends

36. ;
Introduction
Logstash
Kibana
Conclusion
Kibana queries
Example of a kibana query
@fields.syslog_program:"httpd" AND @fields.http_host:"test.example.com" AND @fields.response:"404"
• LLucene query syntax
• SSimple and effective
• PPoint & click web interface
Julien Pivotto Logstash and friends

37. ;
Introduction
Logstash
Kibana
Conclusion
Kibana
Julien Pivotto Logstash and friends

38. ;
Introduction
Logstash
Kibana
Conclusion
Kibana
Julien Pivotto Logstash and friends

39. ;
Introduction
Logstash
Kibana
Conclusion
Kibana
Julien Pivotto Logstash and friends

40. ;
Introduction
Logstash
Kibana
Conclusion
Kibana
Julien Pivotto Logstash and friends

41. ;
Introduction
Logstash
Kibana
Conclusion
Conclusion
• Logstash is a small daemonLogstash is a small daemon
• Simple to package & deploy (jar file)Simple to package & deploy (jar file)
• Scalable thanks to ElasticsearchScalable thanks to Elasticsearch
• Developer friendly thanks to KibanaDeveloper friendly thanks to Kibana
Julien Pivotto Logstash and friends