Syslog is terrible
bouncybouncy.net
bouncybouncy.net
> I don’t particularly care about the argument between plain text versus binary logs. However, if your reasoning for not wanting to use binary logs is because they are opaque and can be corrupted then you should take a close look at your log rotation or archival process. If you are archiving logs by first compressing them using something like gzip, you no longer have plain text log files.
I found quite a few people disliked binary log files due to concerns around their portability. Specifically when they depend on a platform-specific utility to read them. If they are compressed using a standard algorithm, it alleviates concerns about portability.
Well yeah, but that's because I'm already done parsing the log and/or using it for debugging. If I wasn't done with it, I wouldn't have archived it.
But I know that when I pull a 3 year old log out of the archive, I can gunzip it and will still be able to read it, no matter how much the file format has changed since then, and I don't have to dig up a 3 year old magic decoder program (that may only run on one operating system) to be able to see what's in the file.
Anyone who's ran production infra for long enough has written all the regexes, mostly from scratch, mostly badly. Myself included. None of my parsers would have caught the ssh 'root from 8.8.8.8'@server log.
Applications that offer structured logging are superior (in terms of logging), and we need to ensure we build applications that support this going forward.
For those in the python world, https://pypi.python.org/pypi/python-logstash is great. It's somewhat undocumented, but it can be used without logstash, writing python logs with their original arguments to JSON on disk (which we then had an agent ship to logstash, because, disk turns out to be a decent buffer for handling network glitches).
e.g. LOG.error("Invalid user %(user)s", {"user": "Kiall"}) can be logged as "ERROR: Invalid user Kiall", or:
{"message": "Invalid user Kiall", "level": "ERROR", "extras": {"user": "Kiall"}}.
Last time I was writing a logging system the only reason I couldn't write directly to gzipped files was that I couldn't append to a gzipped file using Python (fairly important in logging). It looked like the reason was a Window's specific workaround having to do with seeking from the end of the file.
Oh, but you can rest assured that systemd will never, ever have a breaking change. Lennart Poettering can be trsuted to never do something that would cause anybody any trouble! /s
And that's the problem: systemd is a great idea, and it does a lot of really great things, but the man in charge simply can't be trusted not to break things. It was one thing when he was breaking desktops with PulseAudio; it's an entirely different thing now that his decisions can break servers (how many systems have broken due to nohup no longer doing its job?), and break the ability to retrieve logs over time.
I honestly think that systemd is going to arrive at a good place eventually, and a great deal of credit will belong to Poettering. But it will also cause a great deal of harm before it gets there, and he deserves a great deal of the blame.
For a long while it was flaky and buggy, with audio periodically failing. It was never the end of the world, but it was annoying, and I think it runs decently enough now.
Zero? As far as I know the change that defaulted KillUserProcesses to yes was reverted by every distribution that ships systemd.
The problems discussed in the article could be solved with any structured data format. Logs could be streams of objects encoded using e.g. JSON, bencoding, or protocol buffers.
The article seems to mostly complain about text logs VS structured logs. That is unrelated to syslog because it is simply a log transport mechanism.
Applications should send structured logs. (e.g. JSON messages). The message may be delivered by whatever means (e.g. syslog).
---
Second,
I am surprised that the article doesn't mention the issues with syslog not being defined properly and suffering from interoperability issues.
Truth is: "syslog" refers to multiples, incompatible protocols that evolved other time. See two independent specifications for example:
https://tools.ietf.org/html/rfc5424 and https://tools.ietf.org/html/rfc3164
Applications, libraries and middleware (graylog/ELK/rsyslog/syslog-ng/fluentd) have interoperability issues because of the different syslog protocols. Just because two things are speaking "syslog" doesn't mean they speak the same "syslog".
Switches/routers/appliances cause even more issue because the custom implementations tend to not format messages perfectly for any of the RFC. ^^
> That is unrelated to syslog because it is simply a log transport mechanism.
Only partially, RFC5424 defines the structured data format (that unfortunately is not widely used)
syslog is also the syslog() interface, and as far as I know there is no reference implementation for encoding structured data in the format the RFC describes.
Personally I'd rather it be something like json.
I could patch openssh to log using json, but the chances of that ever getting merged are probably 0.
> See two independent specifications for example:
Well, RFC5424 says it obsoletes 3164 at least. 3164 is just embarrassing, especially the section on timestamps: https://tools.ietf.org/html/rfc3164#section-5.1
Adding a year to the date was "not consistent with the original intent of the order and format of the fields". 5424 at least says "Use this specific form of RFC3339"
> Switches/routers/appliances cause even more issue because the custom implementations tend to not format messages perfectly for any of the RFC
Ugh, I know it. As I said in another comment, we run syslog collection as a service so we receive logs from a large number of different devices not under our direct control.
I found syslog design to be well done about that, ignoring whether it was accidental or not :D
+1 for json. Format messages as JSON for logging. Update/reconfigure applications to write json logs.
The structured data headers of syslog can be used for enriching log messages with metadata (instance id, instance ip, tags, environment, etc...).
The RFC 5xxx design allows to manage the message (application's responsibility) and the metadata (relay's responsibility) separately, so it's nice.
[Note: If ALL messages were JSON logs and the tooling could manipulate json messages directly to add/remove fields, there would be no need for a different metadata channel.]
1. Define an encryption standard, both symmetric and asymmetric (to prevent log tampering). 2. Define a compression standard, with scheduling options similar to cron jobs but defined in syslog.conf. 3. Overhaul the facility field, define some generic ones like "Auth", "Audit", "Kernel" etc and don't hardcode any numeric mappings. Article covers this well. 4. Make CEF the standard format for writing logs to disk or forwarding to other devices. All modern applications and kernels should be writing their logs in CEF format.
We do use rsyslog.. The rates we see aren't terribly high, but it is a fairly large deployment. We have a setup consisting of redundant tcp load balancers fronting redundant syslog relays feeding into additional applications and an archival box.
Basically this example in the RFC, but with a frontend load balancer cluster and additional collectors:
+----------+ +-----+ +---------+
|Originator|---->----|Relay|---->-------|Collector|
| |-+ +-----+ +---| |
+----------+ \ / +---------+
\ +-----+ /
+->--|Relay|-->--/
+-----+
If every client could use rsyslog+relp and things like imfile to send us application logs, the whole system would work a lot better.
It would be great of other systems adopted this.
Does syslog actually support this? I mean, you can send a JSON blob as part of the MSG field, but you still need to send the
PRI VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID
You can't just send something like
{"ts": "2016-09-..", "host": "www1", "proc": "nginx", ..}
We do ship some json logs in rsyslog using imfile, but that's a bit different.