Hacker News Reader

Comment by nailer | Hacker News Reader

The argument re: larger surface for updates is true. Systemd is smaller than the things it replaces, and consistent in itself, but all the logic is in PID 1.

I fail to see a better alternative though.

Old, disparate init systems repeat logic for:

- starting services (the same init script copied 90 times, often badly)

- journalling (unmaintained syslogd facilities means apps all report themselves as local0 and five different fixes for that)

- a billion different 'sentinel / watch' type apps

- various hacks on hacks for dependencies, parallel init, etc.

If you rip out all the redundancy and replace them with something unified, that unified functionality can (and should) exist at low level.

Maybe the argument is that a better way of handling systemd updates is needed?

View on HN

These problems are all solved without destroying the Unix philosophy completely:

starting services - daemontools.

journaling - daemontools

billion different sentinel/watch type apps - daemontools.

dependencies - just don't do it. This is the old age of power sequencers again. It's coupling.

Daemontools is a set of tiny programs that talk to each other, are fully privilege separated and guarantee reliability. Why they can't do this for systemd, I don't know. If you remove the DJB path conventions from it (the major objections), it's the right tool for the job.

> starting services - daemontools

Daemontools is great, but please don't pretend it's a proper general-purpose init system. How do you start a certain service before another one? I suppose you can hack that inside the run script but then you're yet again duplicating logic, badly.

> journaling - daemontools

What? Daemontools provides a nice utility for logging a process stdout/stderr to a file. That's it. That doesn't come close to the problem that journald tries to solve.

> billion different sentinel/watch type apps - daemontools.

This is about the only thing that daemontools solves.

> dependencies - just don't do it. This is the old age of power sequencers again. It's coupling.

And here comes the mythical "coupling is bad" statement while completely ignoring the problem. No, I don't want my NFS daemon to be started before my network connection is up, thank you very much.

Your post shows a complete lack of understanding of the problem.

> Daemontools is great, but please don't pretend it's a proper general-purpose init system. How do you start a certain service before another one? I suppose you can hack that inside the run script but then you're yet again duplicating logic, badly.

I'm not saying its a replacement. The model is suitable as a replacement i.e. inspiration should be taken from it.

What? Daemontools provides a nice utility for logging a process stdout/stderr to a file. That's it. That doesn't come close to the problem that journald tries to solve.

It guarantees that the logs are stamped externally, are committed to disk and that the logs haven't been tampered with by isolating them from other processes and user accounts that can modify or write to them. Don't need journaling on top, just separation of concerns.

Indexing - don't need it. No one does. That's an external problem solved by syslog collectors like Splunk. As someone who deals with up to 500Gb of logs a day, I know my shit. Systemd doesn't solve a thing here. In fact it adds overhead to a solved problem.

And here comes the mythical "coupling is bad" statement while completely ignoring the problem. No, I don't want my NFS daemon to be started before my network connection is up, thank you very much.

Well what the hell does your NFS daemon do when your network goes down, your adapter gets hotplugged after a failure or someone falls over the cable? There is so little of the problem solved by systemd it's unbelievable. The RIGHT solution is to make the processes resilient to failure conditions like this.

> It guarantees that the logs are stamped externally, are committed to disk and that the logs haven't been tampered with by isolating them from other processes and user accounts that can modify or write to them. Don't need journaling on top, just separation of concerns.

What if you want to consolidate all your logs in a single place instead of scattered over many files? What if you want the logs to be shipped to a remote host without storing them locally?

If your answer is syslog: welcome back to the problem.

> Well what the hell does your NFS daemon do when your network goes down, your adapter gets hotplugged after a failure or someone falls over the cable? There is so little of the problem solved by systemd it's unbelievable. The RIGHT solution is to make the processes resilient to failure conditions like this.

That has got nothing to do with the init system. Just because the services themselves can recover from dependencies that temporarily go down, doesn't mean that I want to see tons of useless error messages during startup. I want my NFS daemon to start after the network is up, so that it doesn't bother me with useless "network is down" messages. Recoverability is a completely independent (though desirable) property.

Its not different as both can and must be solved by network daemons listening for network change events. So solving reliability solves the startup issue.

> Indexing - don't need it. No one does. That's an external problem solved by syslog collectors like Splunk. As someone who deals with up to 500Gb of logs a day, I know my shit. Systemd doesn't solve a thing here. In fact it adds overhead to a solved problem.

Does systemd's journal still use a binary format? That is a no-go for me. Inevitably it will get corrupted, and then you rely on the journal reader being able to recover your logs, or fix bugs in it until it does.

I'd be more willing to consider systemd as an alternative if it kept using a human-readable log format, that can be read with cat/tail in a worst-case scenario. If it wants to index it, it can keep an index on the side, no big deal if that gets corrupted/out of date, as it can always be rebuilt.

Yep. It still uses a binary format. An ugly fucker of one too. You can have it pipe to a real syslogd, but at that point, I'd rather just have it send to one and get that ugly monstrosity that is systemd out of the way.

I don't buy this argument. SQLite and PostgreSQL's databases are binary too, but that's ok?

A database can afford to fsync to ensure consistency, and they've been heavily tested to ensure it works properly. And even then its still possible that the file becomes corrupt, if the OS, or the disk lies about fsync: https://www.sqlite.org/howtocorrupt.html

Has journald been tested on how well it copes with sudden reboots, kernel panics, powerloss, etc.?

When something goes wrong you usually want to be able to still read your logs to figure out what happened, and you may not even be able to boot the system properly.

> Has journald been tested on how well it copes with sudden reboots, kernel panics, powerloss, etc.?

Are you assuming it hasn't been tested in these situations just because you don't know?

Databases have been around a lot longer than systemd, so I assume they are better tested than journald in this regard. I don't use systemd - because Debian doesn't use it (yet) - so it was rather a question for those who do use systemd.

I'm not saying that I'd be happy to have a database as journald backend, I'd be just less concerned.

OK, I see. Thanks for the clarification.

systemd's test suite[1] doesn't seem to cover those cases anyway.

[1]: http://cgit.freedesktop.org/systemd/systemd/tree/test

I think it has not been tested in these situations enough. It's not in production anywhere major yet.

The assertion is valid IMHO.

I wouldn't put my system logs into either of those.

> Well what the hell does your NFS daemon do when your network goes down, your adapter gets hotplugged after a failure or someone falls over the cable?

Let's say it crashes or hangs. In that situation, systemd will notice it, clean up, notice that network is not up, wait until it is, and restart it.

Half of what makes systemd so great is that it allows the daemons of the system to be much less robust than it is, and do the right thing so that the entire system will remain robust.

Dependencies between services exist, and for good reason (e.g. various services depending on a database). How do you propose to deal with them?

It's pretty basic software design. A few solutions:

1. Web servers return error pages when databases are unavailable. They don't go down.

2. Postfix doesn't die if your virtual maps database server is not available. It waits patiently.

This isn't rocket science.

I'd rather Postfix fail early, than wait until $TIMEOUT to discover that a maps source isn't available.

I think what parent is trying to say in so many words is "I have no use case for tracking service dependencies, therefore it is pointless and should be cut out". An unfortunately common attitude among technical people.

No. I'm saying that relying on startup sequencing is a bad idea full stop.

That's why systemd does dependency tracking and socket activation instead of sequencing.

It needs to do neither of those as well.

You're certainly entitled to your opinion about dependency tracking being an unnecessary luxury, and I'm sure you will find a number of Linux distributions and/or BSDs within your spartan requirements. Not to mention that you can at least for the time being stay with sysvinit (though this won't solve your issue on Debian, since it reads dependencies via a horrible hack in the comments of the init scripts).

I'm a FreeBSD and Windows person so yes you are right.

sysvinit dependency management is horrible as well so I cannot complain about your point there.

But you're not making a good point. If the network goes down, anything which depends on it will also go down. That's a much stronger guarantee than "let's hope this application handles network failure graciously and will kindly start working again as soon as the network is restored".

So the system will be busy starting and stopping services in case of faulty network, instead of having them sit there patiently while the line continues to flap? Great idea - not sure when Cisco will copycat that one, though...

It doesn't really matter either way, because the symptom will be that you can't send mail; either that's how you find out or you have a proper monitoring solution, in which case it doesn't matter whether Postfix actually died or is patiently waiting for the resource to become available. To sum up, I don't see why dependencies need to exist, it seems better to write daemons in such a way that they can deal with temporary absence of resources.

It does matter. Why are you starting postfix if you have no network? You have plenty of use cases where boot speed is actually important.

On a mail server, it is no great improvement to rapidly boot to a state where mail is not working.

Obviously, this was just an example. If you're making an appliance, boot speed matters. If you're aiming for a fast desktop experience, boot speed matters.

This probably hinges on how you define the word "great," but you can diagnose a nonfunctioning mail server a lot easier if it's actually booted up.

Linux is not a service oriented architecture. That is really more of an economic model than a viable model for engineering fault tolerant designs.

This is a little off topic, but DJB is obviously incredibly talented, but frequently out of step (rightly or wrong) with the general Linux community.

I think it would be excellent if DJB had his own OS - maybe with a Linux kernel, but with a completely different userspace like Android. Specifically:

- the init

- the packaging system and file heirarchy

- the daemons

I'd love to see and use it.

Please suggest something a bit better. Daemon tools doesn't even support limits. As a result, I had to rip it out of a ~2k node cluster if favor of monit.

I'm not suggesting daemontools. I'm suggesting "not systemd" and "something that includes the valuable lessons that daemontools taught us".

The first of which is not to fuck up the Unix philosophy by producing a monolithic pile of junk.

I'm not sure which limits you are talking about. These, or others?

> http://cr.yp.to/daemontools/softlimit.html

You can also use 'runit', which has quite similar design, but comes pre-packaged for a lot of systems, and can run as init, too (only if you want to).

> This is the old age of power sequencers again. It's coupling.

Is this referring to power transmission or something?

No. Back in the bad old days of big Unix machines and mainframes we used power sequencers to bring physical bits of hardware up in order. This entire problem has now moved into the software space. With hardware, dependencies were painful, expensive and relied on lots of voodoo and dependency resolution. The same with software. Eventually the hardware manufacturers and the OS people got together and threw this out of the window.

Now Linux is about to fail to learn from those mistakes like it failed to learn about audio subsystems, firewalls, network configuration, desktop environments, graphics architecture etc.

Maybe this is naive, but can't that repeated logic be put into a shared library (or a shared helper script)? A common, shared service for starting services reliably, logging them coherently, managing service dependencies, and restarting failed services seems like a great idea. But PID 1 doesn't seem like the right place for them.

You're right. But there are several concerns here. The whole thing is a bunch of concerns crudely bundled into one concern inside a single package in systemd.

This is already done in another operating system where it is successfully broken down into components which are exposed through a system-wide common RPC and component architecture so notifications and instructions can be moved around seamlessly. The whole thing is covered by ACLs, privilege separation, has immutable logging and a configuration database to drive it all. Oh and it's about 20 years old now.

And it works really well.

I'll let you guess the vendor (and no it's not Apple).

They can, and init systems already do that. Init scripts import a common library with functions for starting and stopping daemons etc. However the result is still a big mess. Libraries are not frameworks, so you still end up with tons of boilerplate code everywhere. Take a look at /etc/init.d/* and see whether you can make sense of things at a glance.

Furthermore, SysV init is not designed for service supervision, so SysV init scripts cannot implement that functionality.

Reply on news.ycombinator.com