Holy balls, talk about going out of your way to avoid syslog.
* Freeform text is a terrible way to track system events.
* Periodically rotated flat files are not a great way to store log information.
* Goofy little UDP messages are not a good way to convey system events
* The syslog PRI field dates back to when we exchanged messages with UUCP.
I could keep going, but since you're just going to reply with "lolwut umad?", I'll leave it at that.
Out of pure curiosity, what do you see as the tool most likely to displace syslog in the future? Is there any alternative available that fixes most of these problems without rolling your own from pieces and parts?
And I'll add it tends to ship in a horrible default configuration with events scattered randomly over multiple files, no safe-guards against filling up the disk and no safeguards to ensure the stupid daemon is actually running.
However...
Freeform text is a terrible way to track system events.
Nothing stops you from logging structured text.
Periodically rotated flat files are not a great way to store log information.
Modern syslog daemons will write to pretty much anything you want.
Goofy little UDP messages are not a good way to convey system events
Modern syslog daemons offer tcp transport. Some even try to offer some delivery guarantees (disk-backed spool), although personally I wouldn't rely on that for truly critical stuff.
The syslog PRI field dates back to when we exchanged messages with UUCP.
Thanks, I always wondered where those were from...
And, well, you forgot a couple bullets:
* syslog() is available everywhere, out of the box
* It's trivial to move from file-based logging to syslog
* We have mature syslog-daemons that dispatch events pretty reliably
* Unless you're facebook you probably don't need anything more fancy.
So, I'd say syslog gets the job done quite well, as long as you don't mistake it for a message queue.
It's not just that logs tend not to be structured, it's that the metadata is all inband. And syslog in particular exacerbates the problem by decoupling logging clients from log storage policy: the log generator has no idea if its syslog has a super-smart storage policy and so has to assume everything needs fine grained human readable timestamps, a custom facility/severity notation, the proctitle and pid, and so on.
2.
Most syslog daemons will store anywhere. Sure. The last syslog daemon I actually read had stack overflows in it, so it's been awhile for me. But if you've got syslog writing to a real store, what value is syslog adding? The standardized network protocol? Redis has a trivial network protocol too.
3.
The issue with syslog facilities isn't that the "UUCP" facility harms anything directly; it's that one the few bits of out-of-band metadata syslog truly offers forces applications to decide whether they're "kern" or "daemon" or "local2". Whose application actually breaks down cleanly into syslog facilities?
4.
I dispute that syslog is "good enough if you're not Facebook". I think you do too: you probably aren't doing all your metrics stuff with syslog logs; you probably aren't even getting your web stats though syslog. You may have even delegated your web logs to client side Javascript, because that is how bad text logs stored in flat files are to work with.
Exactly. Every piece of software under the sun either logs to syslog by default or has a config-switch/patch to make it so.
Yes, syslog the protocol is pretty nasty. Yes, syslog daemons are pretty nasty. But they get the job done and you are free to put anything (including redis) at the end of the pipe.
Coping with syslog is by far easier than trying to make all "legacy software" including the kernel speak something else entirely.
But if you control it, why bother with syslog? Syslog is a piece of junk. Don't bother. Any ad hoc scheme you come up with that uses a real backend store will be better than syslog.
I'd ask this backwards: Why bother with a homegrown solution when syslog exists and gets the job done?
A home-grown solution takes at least a couple weeks to stabilize, likely much longer before the last bugs are ironed out. Syslog takes about a day to beat into shape.
Any ad hoc scheme you come up with that uses a real backend store will be better than syslog.
You know better than that.
Shipping messages reliably is a surprisingly tedious problem. First you realize you need a disk-spool. Then you realize that spool should be size-capped anyhow. Eventually your boss says you need a network topology more complex than A->B for some idiotic but inevitable reason.
And then next month you run into some redis limitation and realize some kind of datastore abstraction would have been a better idea to begin with. Hmm, perhaps dump to plain-text files until we figure that one out?
See the pattern here?
At the end you have reinvented syslog. Sure, yours may be nicer or at least different.
But that's a whole lot of work to avoid something that, despite all its warts, already works.
Pretty much any other time I've had to work with it, I've wished for something better, so I, for one, am very happy by the thought of people starting to "go out of their way to avoid syslog".