Rsyslog vs Systemd Journal Presentation

rsyslog vs journal?
Rainer Gerhards

Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany
Me & the Talk
• Rainer Gerhards
▫ Data center guy
▫ Involved 15+ years in logging
▫ Founded rsyslog in 2003
• The talk
▫ Will rsyslog fight the journal?
▫ Some history on journal-like system
▫ Ways of integration
▫ How to do things the journal announcement
claimed as impossible

Does journal replace syslog?
• The initial announcement sounded a bit in that
way, or was at least interpreted by most
(including me) in that direction.
• Looking at how things have evolved
▫ There of course is overlap between both systems
▫ But there are also (large) regions that do not
overlap
• This is not a new situation, there is some history
lesson...

Windows Event Log!
• The Windows Event Log is in many ways similar
to systemd journal
▫ Binary database with rollover and fast access time
▫ uses a simple structured format that captures core
metadata items (like timestamps, user IDs, …)
▫ uses unique identifiers for different types of log
messages
▫ Files are especially secured by OS

Event Log History
• Introduced with Windows NT 3.1 in 1993
• Greatly enhanced in 2007, starting with
Windows Vista
• Originally single-computer only
• Now provides network functionality
▫ EventLog-to-EventLog push and pull
subscriptions
▫ Can be used to setup log forwarding in the
enterprise

So what does history tell us?
• If such a system can totally replace syslog, there
should be no syslog on Windows at all – and
never have been.
• Well... there are ample of applications
▫ WinSyslog (initial version by me, 1996)
▫ Kiwi Syslog (Solarwinds)
▫ EventReporter (first ever Windows-to-syslog tool,
1997)
▫ Snare
▫ and many more!

Obviously, there must be some
need to syslog technology...
• Face it: syslog is the lingua franca of network
event logging.
▫ If you want to process messages from different
sources, chances are high you will need it.
▫ Even if not syslog (protocol) is used, you usually
need some common denominator
 e.g. Linux does not understand native Windows
EventLog
 Windows neither does understand native journal

A key problem solved by syslog
• You want to integrate all of your systems into a
consolidated log
• This either means
▫ A common protocol
▫ A system that is capable of processing multiple
protocols and somehow “normalize” them
• Syslog is ubiquitous – because a basic client is
dumb easy to implement!

Window as a sender...
• Early days: missing network functionality was a
problem; brought up the idea of Event Log
forwarding
• Big customers quickly adopted that for
integration into their management system
• Today's hot topics:
▫ local filtering and preprocessing
▫ Ability to extract and properly express OS objects
▫ Support all Windows capabilities
▫ Secure protocol choices

Windows as a receiver...
• Windows acts as syslog server
• Messages are written to
▫ Local files
▫ Windows Event Log (!)
▫ Some other processing (like alerting)
• Typical deployment scenario for SOHO
• But some large Windows-only shops also use it
for integration of non-Windows sources

Why I am talking so much about
Windows?
• As I said, I see strong similarities between
journal and Windows Event Log
• Except that journal has much more quickly
gotten some network functionality
• So my best guess is that deployments and end-
user needs will evolve into mostly the same
directions

Journal vs. Syslog:
low end systems
• Usually users of these machines are not at all
interested in logging
• Journal is very convenient as a troubleshooting
tool
• Works perfect on personal desktop & notebook
• Rsyslog will be needed by some users to
integrate e.g. their DSL router's messages into
the journal

Journal vs. Rsyslog:
enterprise systems
• Impossible to manage without any syslog
• Journal integrated as another event source
▫ Journal-centric
 As much as possible is done with journal
 Integration happens at central head server(s)
▫ Syslog-centric
 Journal is used only as much as unavoidable
 Each machine runs rsyslog and forwards events
▫ Mode depends on end-user's philosophy

How did the journal affect the
rsyslog project?
• Obviously, we expect less presence on low-end
systems
• So we re-focussed the project
▫ Previously low-end and enterprise needs were
equal peers
▫ Now strong focus on enterprise
• The logging world at large got benefit as
suddenly everyone was interested in logging –
which also helps rsyslog!

What have we done to integrate
with the journal?
• Module omjournal
▫ Provides ability to store messages into the journal
▫ Traditional syslog, text files, ...
▫ Caters for the low-end use case
• Module imjournal
▫ Provides ability to pull messages off the journal,
just as another event source
▫ Contributed by Red Hat
▫ Caters for the enterprise use case

Integrating syslog Data into the
journal (SOHO env)
/* first, we make sure all necessary modules are present: */
module(load="imudp") # input module for UDP syslog
module(load="omjournal") # output module for journal
/* then, define the actual server that listens to the
* router. Note that 514 is the default port for UDP syslog.
*/
input(type="imudp" port="514" ruleset="writeToJournal")
/* inside that ruleset, we just write data to the journal: */
ruleset(name="writeToJournal") {
action(type="omjournal")
}

Integrating journal data into syslog
module(load="imjournal" PersistStateInterval="100" StateFile="/path/to/file")
module(load="mmjsonparse") #load mmjsonparse module for structured logs
$template CEETemplate,"%TIMESTAMP% %HOSTNAME% %syslogtag% @cee: %$!all-
json%n" #template for messages
*.* :mmjsonparse:
*.* /var/log/ceelog;CEETemplate
• Necessary to obtain extended journal properties
• If not needed, regular system log socket can be
used

Why is it simple to integrate the
journal?
• Rsyslog is actually
▫ A message router
▫ With dynamically loadable inputs and outputs
▫ Highly configurable
• So, journal support is as easy as adding some
new inputs and outputs!
• The rest of the plumbing is already there.

rsyslog Architecture
journal
Network
(e.g.TCP)
/dev/log file
Database
Remote
system
Parsers
For-
matter
Rules
&
Filters

“String” Handling in rsyslog
Message String
Parser
Dictionary of Name/Value Pairs
Formatter
Output String
“special”
Outputs
Modifi-
cation Mod

Now let's look at some
“impossible” things
• The original journal paper claimed that syslog is
▫ Seriously broken
▫ Cannot provide some important features
• I'll show how to do these “impossible” things
▫ Based on 2011 technology
▫ And on current one (v7.4)

Log File Manipulation Protection
• The traditional approach is to ship logs off the
machine, to a central and highly secured system
• Keeping them on a system that is “easily
compromised” is asking for trouble.
• Problem is that local secrets can always be
compromised
• In rsyslog 7.4, we address these problems via log
signatures and encryption...

Signed Log Records
• In 2011, there was no good solution (and
journal's solution was also not good)
• Things have evolved since them
▫ Journal got “forward secure sealing”
▫ Rsyslog
 got a crypto provider interface and a provider for
“Keyless Signature Infrastructure” (KSI)
 Hash chain for log record is created, and key hashes
are chained in a global hash chain, which provides
signature & timestamp

Signing via Hash Chains...
• Very rough sample (actually Merkle trees!)
• No local secret!
• Consider “chain layer” to be operated on a
schedule (timer ticks!)
Source: http://en.wikipedia.org/wiki/File:Hashlink_timestamping.svg

Activating Log Signing
action(type="omfile"
file="/var/log/logfile"
sig.provider="gt"
sig.keepTreeHashes="on"
sig.keepRecordHashes="on")
• Parameters except sig.provider are optional
• Writes
▫ regular log file
▫ plus signature file (*.gtsig)

Log File Encryption
• Crypto-Provider interface recently added
• As well as a libgcrypt-based crypto provider
• Symmetric cryptoraphy, all ciphers & modes
supported by libgcrypt
• Key can come from
▫ Config param (testing only, pls!)
▫ File
▫ Script (interface for advanced key exchange
options)

Activating Log Encryption
action(type="omfile"
file="/var/log/logfile"
cry.provider="gcry"
cry.keyprogram=”/path/to/binary”)
• Addtl Parameters for ciphers, etc...
• Writes
▫ regular log file, encrypted
▫ plus encryption info file (*.encinfo)
• Works in conjunction with signatures
• In 7.5 extended to rsyslog disk queues!

Syslog Network Processing
• Original Journal accouncement missed
improvments and talked only about UDP syslog
• We have
▫ TCP & TLS support (RFC5425, 2009)
▫ Mutual authentication & authorization
▫ Multiple hops
▫ Buffered send queues, even with disk buffers
▫ Rsyslog can utilize other protocols as well (RELP,
SNMP)!

Sample: TLS-encrypted for-
warding using a buffer queue
$DefaultNetstreamDriver gtls # make gtls driver the default
# certificate files
$DefaultNetstreamDriverCAFile /rsyslog/protected/ca.pem
$DefaultNetstreamDriverCertFile /rsyslog/protected/ma-cert.pem
$DefaultNetstreamDriverKeyFile /rsyslog/protected/ma-key.pem
# authorization
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer central.example.net
$ActionSendStreamDriverMode 1 # run driver in TLS-only mode
# Queue
$ActionQueueType LinkedList # use own queue
$ActionQueueFileName fwq # set file name, enable disk mode
*.* @@central.example.net:10514 # forward to remote server

Indexed Store
• Core idea: connect to existing “life stores”, do
not push that part into the syslogd
▫ Actually one of the core requirements that started
rsyslog (MySQL, ~2004)!
• 2011: various relational databases
• Today also
▫ Document based databases (MongoDB,...)
▫ Elasticsearch
▫ Journal DB

Rate-Limiting
• 2011
▫ Repeated message reduction
($RepeatedMsgReduction on)
▫ Output throttling
• V7, additionally
▫ Repeated message processing on a per-input basis
▫ Object-based rate limiters (n messages within s
seconds)

Log compression
• Log files can be zip compressed by specifying the
“ZipLevel” parameter
• Of course, accessing compressed log records
requires more processing time.
• Today, we also have experimental code for log
transfer compression (directly built into the
protocol/output modules).

Message authentication and
Metadata availablility
• Rsyslog uses the same SCM_CREDENTIALS
facility that journal does
• And in both cases it can be faked – as journal
demonstrates when it actually fakes it on the
system log socket ;)
• The volume of metadata available has been
increased starting in 2012
• Total authenticity requires signatures at
the original originator level (each app),
what currently is impossible in the *nix
framework.

Free-Formedness of Log Records
• Traditional syslog messages are much like free-
form text
• Today, we see the same for typical journal
messages
• There are a couple of standardization efforts
underway to provided structured logging
• Project lumberjack (lead by Red Hat) provides
JSON-based structured logs

Unstructured Text Log Duality
• If a log format does not support freeform-text, it
is not used (at least not more than one can avoid
to...)
• If it supports freeform-text (among others), that
freeform-text will be abused
• → unstructured logs won't go away!
• We've seen this in Windows Event Log and looks
much the same for journal.

Converting Free-Text Messages
via mmnormalize
• Uses a “sample rule base”
▫ One sample for each expected message type
▫ Sample contains text (for matching) and property
descriptions (like IPv4 Address, char-matches, …)
▫ If sample matches, corresponding properties are
extracted
▫ Special parser for iptables
• Very fast algorithm (much faster than regex)
• Based on liblognorm (which can also be used in
other programs to gain this functionality!)

Timestamp format
• No year, no timezone
• ... just because distros turn it off.
• Remove “$ActionFileDefaultTemplate
RSYSLOG_TraditionalFileFormat” from
rsyslog.conf to get rsyslog's default high-
precision RFC5424 timestamp
• Some tools may have problems with that, but
can't be too bad – some Distros use the default
format

Questions?
• rgerhards@adiscon.com
• http://blog.gerhards.net
• http://www.rsyslog.com
• http://www.adiscon.com
• Associated paper is
available on SlideShare.

Rsyslog vs Systemd Journal Presentation

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Rsyslog vs Systemd Journal Presentation

Semelhante a Rsyslog vs Systemd Journal Presentation (20)

Mais de Rainer Gerhards

Mais de Rainer Gerhards (10)

Último

Último (20)

Rsyslog vs Systemd Journal Presentation