Comment by SwellJoe | Hacker News Reader

Comment by SwellJoe | Hacker News Reader

Parent Full thread

On the logging question, does the ForwardToSyslog option not do what you'd like?

Most components can be replaced (the journal isn't one of them, AFAIK, but most others can, and the journal can be configured to send things to a plain text log). Personally, I don't find the binary log to be that problematic. It is taking some getting used to, of course. 20+ years of muscle memory for grepping and awking and cutting text log files is hard to overcome, but the amount of help the system provides is nicer. It's better documented and better at guiding the user than the old way, IMHO.

Which brings to mind something I like about systemd becoming standard that rarely gets discussed, I think: It brings a level of consistency that some other UNIX systems have had for a long time; the BSDs, for instance, have the feeling of all of the pieces having been designed to work together. Linux has never had that feeling; but with systemd, a large amount of Linux real estate now feels that way. The documentation correctly references other components docs. The way various components work together is reasonably documented because they were built at roughly the same time by roughly the same people (or at least people who interacted during the design), so the interfaces are clearly delineated. That's cool, I think.

So, sure, I'm having to learn new stuff every time I interact with systemd, and that's frustrating. I know Linux like the back of my hand, so when I find myself having to read the docs for even simple stuff, I get grumpy. But, I get over it. It's gonna be fine, and we'll all get through this learning curve.

> On the logging question, does the ForwardToSyslog option not do what you'd like?

Of course not. Now you have two logging services. Hardly a replacement.

So, to be clear, it is systemd itself that expects to be able to spit its logs to journald. Other services (most of the rest of a system) can log to whatever target it wants, including plain text files or a running syslog.

The idea is that systemd wanted to not implement a bunch of different logging APIs within systemd itself. They wanted one logging API across all of the systemd components; and, then, if the end user needs logs to some other target, they use one utility (journald) to spit it out to that other target.

I believe the thinking that went into this decision was sound. The alternative would be for every component in systemd to know about multiple logging targets, which would be exactly what people are accusing systemd of being (but isn't, in this case): bloated. systemd and all of its components log one way, allowing for a very simple logging code path. journald allows you to use those logs in many ways.

So, either, systemd would need to expand to include support for a bunch of different logging mechanisms, or it would support one simple logging API and provide a way to export those logs to other targets.

Are you here arguing that systemd should be made larger and more complicated, in order to support other kinds of logging (like syslog and plain text files) without the help of journald?

To me, this looks like modularization and componentization at a reasonable level of abstraction. It simplifies the logging code for every component, while also providing a means for end users to consume the logs in the ways they prefer, including plain text.

Your reasoning is invalid. It would be just enough if systemd supported syslog, with a mechanism for buffering logs initially until the syslog starts, which systemd already needs and has anyway. It was not a choice between new, custom service and making things complicated by supporting every possible log service.

It's not modularization. It's scope creep in pure form.

There's an argument to be made for picking an existing standard. I probably would have, were I developing something like systemd, because I'm lazy. But, the path they've chosen is more flexible than that, and has some other benefits. syslog is a very old standard, at this point, and has many limitations and issues, particularly in a world where a deployment might involve hundreds of instances of a service.

Lennart wrote a pretty good summary of why journald exists, and I think it holds up to scrutiny pretty well:

https://docs.google.com/document/pub?id=1IC9yOXj7j6cdLLxWEBA...

One could argue about what the right "one log" target is, but I doubt one would come up with syslog. syslog-ng and rsyslog were incremental improvements, and solve some of the issues, but they're 18 and 12 years old, respectively, and both pre-date containers, widespread cloud computing (AWS came in 2006), service-based architectures, etc. When they were new, we still had servers that did one thing.

One can also argue about whether journald is the right implementation, even if we acknowledge that a new standard is needed, we don't have to agree that journald is the right one. But, I wasn't writing the code, and what they came up with is pretty good. I don't have any major complaints about it, except that I have to learn some new stuff.

So, I respect the argument that they should have chosen an existing standard for logging, but I also see the reasons why they chose to build something new. I think if I'd been involved in the decision, I probably would have come down on the side of journald, in the end (but would have needed some convincing).

> Lennart wrote a pretty good summary of why journald exists [...]

Of the reasons he wrote it, half of them is not met by journald, and the other half could be implemented without much trouble on top of syslog. Hard to call it "holding up to scrutiny pretty well".

And I fail to see how journald is better than syslog in the age of containers.

I think we'll just have to agree to disagree, then. You don't like systemd or journald, and that's fine. To me, it looks like systemd and journald solve the problems they set out to solve in a reasonable way; I don't like everything about it, but I didn't like a lot of things about initscripts and syslog, either, and think we've moved up in the world by switching.

Hopefully, you'll be able to direct your energies into maintaining one of the distros that has chosen to stick with the old way of doing things (Devuan seems to be the most promising one coming up, but I guess there are several others). I'm not opposed to folks having choices, as long as I don't have to maintain them.

To detach from discussion at this point is fine, I guess. But please entertain my question about containers. I'm yet to try those out, be it Docker or some other packaging solution[#], so I'd like to know beforehand what is different with them in logging than what I usually expect from regular log centralization.

[#] LXC/libvirt type containers I see as merely cheap VMs, though I like them very much.

I think me backing off is the only useful thing for me to do. We both seem pretty convinced about what the right solution to these problems is, and neither seems to be budging. I don't really think either is right or wrong; I have a preference and so do you.

As for Docker, you can continue to use the logging tools you're used to, or you can have it output to the journal. Docker supports many logging drivers: https://docs.docker.com/engine/admin/logging/overview/

If you draw the line there you're probably running a dozen "logging services" on any given system.

Reply on news.ycombinator.com