Syslog: Complete System Administrator Guide | Hacker News Reader

Syslog: Complete System Administrator Guide

devconnected.com

274 pointsSCHKN35 comments

It seems worth noting that the "Syslog message format" was introduced in RFC 5424; the article cites RFC 3164 for message delivery (and should be citing RFC 5424 based on the format).

RFC 5424 defines a more formal and more complete version of the protocol. NetBSD understands and emits it, FreeBSD is still in the process of gaining support for it and OpenBSD seems to be perfectly content sticking to traditional BSD syslog as recorded in RFC 3164.

Are there any systems apart from hardware appliances that actually care about the "new" syslog format? I'm running into custom aggregator agents, fluentd, and trivial forwarders everywhere. I don't think I've seen structured syslog or the extended metadata properly used anywhere.

Since we're mentioning some of that newer tech, it might bear bringing up things like http://jsonlines.org for log shipping. If your destination is an ELK stack, this lets you do some neat stuff like adding arbitrary fields in the log shipper and then filter on them in the backend.

Yes, most SIEMs and log management systems support the new format, even if they also push their own proprietary format.

And let's not forget that there are non-standard "syslog" implementations that were being used for decades before RFC3164 happened. Cisco devices and AIX are the first two big ones that come to mind.

It's also worth mentioning that newlines are significant in TCP syslog (for batching log messages) but not in UDP syslog (because of frame size limits)

> It's also worth mentioning that newlines are significant in TCP syslog (for batching log messages) but not in UDP syslog (because of frame size limits)

More because of the D in UDP, I believe. UDP is a packet (datagram) protocol; TCP is a stream protocol.

> And let's not forget that there are non-standard "syslog" implementations that were being used for decades before RFC3164 happened.

totally agree these exist and may vary wildly - many of these are often proprietary forks of the BSD version in some level of (un)maintained state, FWIW

this is a great summary of the protocol. I recently needed to debug some weird router behavior. An rsyslog change saved me from having to deal with the crumby router debugging GUI. All that said journald really is clutch.

sudo journalctl -o json -f | jq .

Ooh, I didn't know that journalctl could output json.

Thanks!

Note that most current Linux distributions use journald, rather than syslog. Journald has plugins for the syslog wire protocol though. Personally I prefer app names vs LOCAL4 or the other syslog facilities (UUCP is still in there) but YMMV.

I was actually going to ask a question around this. On Ubuntu systems, I see both journald and rsyslog. How are they related? Is journald forwarding to rsyslog?

Yes. journald uses HTTP for remoting, but I think Ubuntu is set up for syslog compatibility. To set up journald remoting: https://serverfault.com/questions/758244/how-to-configure-sy...

Correcting self: I misread your comment as saying "Is journald forwarding to syslog?" which is accurate. "Is journald forwarding to rsyslog?" (note the 'r') is not accurate.

There are two options. One which works with any syslog daemon: you enable ForwardToSyslog, which creates a socket that can be used instead of /dev/log. Or another for specific daemons: rsyslog and some others can read messages directly from the journal themselves. Option 2 is the default on new systems.

/dev/log is probably a symlink to /run/systemd/journal/dev-log on your system. 'systemctl list-sockets' will show you that the socket is part of the systemd-journald-dev-log.socket unit. So when a program calls the syslog(3) function, it ends up sending the message to journald.

If configured to forward messages to syslog, I think systemd does so via the /run/systemd/journal/syslog socket, which is part of syslog.socket. But I believe rsyslog will these days read messages directly out of the journal rather than relying on journald to forward them.

You've got it right besides the last sentence. Copying from systemd-journald-dev-log.socket comment:

Note that journald internally uses the this socket both for receiving syslog messages, and for forwarding them to any other syslog

edit: formatting

That's a but confusing and I don't know how it would work. But last time I looked into how ForwardToSyslog= worked I discovered it was disabled in the default build configuration these days because rsyslog prefers to ingest from the journal rather than being fed messages via a socket...

See https://unix.stackexchange.com/a/294206/5132 .

Unfortunately journald leaves out things like MDC information as well, which usually means JSON logging and trying to avoid broken JSON objects due to log message truncation in some part of the chain.

Sorry, what does MDC mean in this context? Thanks.

"Message digest code" (a hash) I suppose.

Does the PRI formula seem nonsensical to anyone? Facility number * 8 + severity level = PRI. At first looking at the example, I was thinking it would be an easy way to look at the last digit to determine severity (0-7), but since the result of facility number * 8 could contain any final digit, so that's a no go. Since severity is 0-7, and one multiplies facility by 8, no numbers would overlap, so one could eventually get used to the ranges and understand that, for example, PRIs 32-39 have something to do with auth (auth is facility 4, times 8 = 32, + 0 for emergency, up to +7 for debug), with 32 being an emergency and 39 being debug, but it seems unintuitive unless you're a sysadmin that sees these frequently.

It seems like simply concating the facility and severity level would give a much more intuitive at a glance understanding of what's going on, and make it easier to awk all, say, emergency severity levels since they'd all end in 0, or awk for all cron facilities, as they would start with 09, or all FTP errors (FTP facility 11 + error 3, 113). Range would be 000 (for kernel, emergency) to 237 (for local 23, debug). Of course, I don't know anything about syslogs except what I learned in this article, so maybe this is totally off base.

Bitwise operations: PRI = (facility << 3) | level

In term of readability, it becomes clear if you use octal and not decimal representation: the last digit is security level whereas the others are facility number.

grepthisab’s comment is about the textual representation, which uses a decimal number < 192.

A bit to my surprise, the textual representation also uses that for the ‘on the wire’ protocol (and that isn’t to keep the format a text format. The ‘MSG’ part of a syslog message can (but SHOULD NOT) contain arbitrary byte sequences (https://tools.ietf.org/html/rfc5424#section-6.4). I would have expected this to be packed in a single byte there.

So, we have potentially binary messages, _and_ a syslog parser must know quite a bit of Unicode. Fun :-)

I think it's because they aren't really supposed to be human readable. It's trivial for an application to do (PRI % 8) to get the severity, and ((PRI - severity) / 8) to get the facility number. I'd assume that's why severity only goes to 7.

This could make sense, but I don't see why being human readable wouldn't necessarily be a benefit. I do a lot of log shaping with sed, grep, awk, etc. and human readability has been a plus for me on things, but I may not be the norm.

Regarding the second point, I believe the 7 is so ranges don't overlap. So for example facility X, severity 7 doesn't equal facility X + 1, severity 0.

> I don't see why being human readable wouldn't necessarily be a benefit

That's the wire protocol, right? When rendered, this might show up as "2019-08-05T08:12:22Z <4.3> hostname ..."

> I'd assume that's why severity only goes to 7

I think it may be the other way around: with severity going up to 7, the multiplication factor was chosen to be 8.

For the people down voting this post, please see https://xkcd.com/1053/

The formula is like that because there are 3 bits of information in the Severity Level field: it has 8 values (0->7), and 2^3 = 8.

Multiplying the Facility Number by 8 (or, equivalently, by 2^3 or by left shifting the number by 3), frees up the bottom 3 bits.

This is the same operation as you are suggesting to do in base-10: multiply by 10^3 to free up the bottom 3 digits, but for base-2 (i.e. in binary).

When the Facility Number has been moved out of the bottom 3 bits you can add in the Severity Level and get a combined value for both.

It's true that it's not very intuitive when they then render the resulting number in base-10 / decimal, however, it does get you the most compact encoding given the size of the two fields.

As another poster says, if the number had been rendered in octal (base-8) then the Severity Level would be the right-most number in the series.

Hey thanks this is really helpful, appreciate you taking the time to explain.

Interesting TCP is recommended with pain old Syslog. I've seen that take out systems due to not being able to dequeue. There are better protocols that you can use for reliable delivery

What would be an alternative option?

There's RELP[1] but if you can don't use syslog's protocol. You could use something like the beat protocol (if using ELK stack) or just a plain old message queue. Things like logstash (and fluentd etc) can have multiple input/output targets.

You'd have something like filebeat or fluentd reading the logs locally and then shipping via that protocol to a central system where they'd be ingested. For application logging, definitely use structured data (like JSON, for example) over log-lines. It's easier to parse in the long-run.

1: https://en.wikipedia.org/wiki/Reliable_Event_Logging_Protoco...

RELP also uses TCP, so how would that help in a situation where Syslog over TCP doesn't work?

REPL != Syslog TCP. They're different things. It's not TCP that was the issue but the implementation of syslog on top.

Reply on news.ycombinator.com