Hacker News Reader

> Compared to sysv-init this is a regression, because init scripts usually ran sequentially and also didn't involve unreliable systemd-udev events.

I find this really odd because we spent decades with that exact class of problems on SysV init, in part due to the primitive dependency model and the lack of standard mechanisms to launch daemons or react to events meant so many different reinventions of that particular wheel (need Apache to wait for NFS mounts? Here’s another shell kludge to maintain!).

Switching to systemd gave a single, simple answer for all of that: use the init system. I don’t know what services you manage but for the mostly Debian/RHEL environments I’ve worked on your description more fits the before state than after in my experience.

Apache waiting for NFS mounts is still not solved. Now Apache may depend on a file system, but it still cannot depend on "the LDAP is available so you can resolve the UIDs on that NFS". So we are a tiny step ahead maybe. But the problem isn't solved at all, and back in the day we just solved it by reordering Apache manually all to the back, maybe with a "check that UIDs are resolvable"-script in front. systemd can't do that (except if you allow "evil" shell-scripts in bootup again...).

I don't see why you couldn't do that in systemd (or any other init), it's not hard to make a small unit that simply runs a shell script.

But it is a mess. You'll need 2 files for that unit (itself and the shellscript) plus changes to all necessary dependencies to put it in the right place. A sysv-init script is just edited in the right place and you are done.

> A sysv-init script is just edited in the right place and you are done.

If you never intended to update the package, that owns the sysv-init script, that is.

If you do, you will find out at update time, that you have to manually play with 3-way diff and hope, that the init script didn't change too much. You are never done.

With systemd, you have an override, you don't change the original unit. It will update cleanly.

I think the implication is that "systemd as designed" cannot do that, and shell scripts are a hacked-in solution. I disagree with this implication because the fact that systemd left that door open is enough for me, but I get where they're coming from.

1. We didn’t “solve” that back in the SysV days by reordering - mostly we just lived with the occasional race conditions and restarted it.

2. You could use systemd’s health check or pre-exec to handle prerequisite checks like this if you can’t use the dependency mechanism for some reason (there’s no reason why you couldn’t have a custom type=oneshot service set as a requirement for Apache).

There’s nothing “evil” about scripting like this - the thing which makes me tired of SysV is the limited standard feature set (daemonization, logging, restarts, dependencies, event handling, etc.) and challenges coordinating across distributions for things like overrides. The wheels fall off shell scripting when you need to deal with that much complexity and do so in a portable manner with a bunch of shell code you don’t control - a health check is far less tedious to deal with, especially since you have a ton of functionality in systemd which avoids the need to handle the hard parts.

In that sense, systemd doesn't solve anything for the same reason sysv-init doesn't. All those preconditions in systemd will still either time out or never finish.

The point is, back then I just moved the apache startup script to back, added a wait-and-check loop in that script and had everything in one place and done.

With systemd I have to rely on a dependency on some notoriously broken network-up-target, add a separate script with my wait-and-check loop to the apache unit, add a dependency on an NFS mount or maybe autofs and then hope I didnt forget something. Then notice that autofs units are usually broken because they exit too early (before they mount anything) and add a "sleep 10" unit in between. Now I'm at more complexity, a shellscript, multiple files and still not ahead in functionality and reliability. Progress!

> In that sense, systemd doesn't solve anything for the same reason sysv-init doesn't. All those preconditions in systemd will still either time out or never finish.

No, they will run until the configured timeout and be restarted following your configured restart policy. Once that's happened, all dependent services will automatically be started.

> The point is, back then I just moved the apache startup script to back, added a wait-and-check loop in that script and had everything in one place and done.

Nothing prevents you from putting a wait loop in the Apache systemd unit, either — in fact, it'll be a lot easier because you can use an override rather than needing to modify a packaged file which will be clobbered by updates.

> Now I'm at more complexity, a shellscript, multiple files and still not ahead in functionality and reliability. Progress!

It's interesting because you keep describing what sounds like the SysV experience, which I haven't seen since switching to Upstart/systemd over a decade ago.

At best, ordered startup should be optional, since boot otherwise takes too long on workstations.

It would be nice to have hard errors when dependencies are not explicitly declared. For example, if you don’t declare dependency on the network, your service gets put in a sandbox where it can’t access the network at all. In practice, that kind of sandboxing sounds difficult.

My personal experience is that it is fairly straightforward to debug races in startup. Some service fails, you look in the log and see that it has a dependency that hasn’t started, and then add that missing dependency to the service’s declaration. Just speaking from personal experience—this doesn’t come up especially often and doesn’t require a large amount of time to fix, and the benefits of fast startup save you much more time than the extra debugging costs you.

I rarely boot my workstation, certainly no more than once a month (I don't bother with every new kernel update)

I reboot my workstation about once a month, and I couldn't care less how long it takes. What I do care about is that my workstation actually reboots predictably. If it needs manual intervention when I'm traveling, I'm screwed.

My POST takes longer than my serialized init scripts on my workstation.

> Compared to sysv-init this is a regression, because init scripts usually ran sequentially and also didn't involve unreliable systemd-udev events.

This goes both ways - compared to sysv-init we finally get a good way to manage services in response to device changes.

It depends on the application. For laptops, sure, hotplugging stuff is important. For servers, not so much, reliability is far more important than getting your USB-stick automatically recognized. systemd is heavily skewed towards the unreliable but fast and dynamic laptop usecase.

A server won't have many udev messages happening after boot, (Past the drive and network cards) so it shouldn't hold up and services either. Have you got specific situations in mind that are not reliable?

Network cards aren't simply just cards anymore. Often you get multiple interfaces depending on each other, e.g. bridges depending on multiple ethernet interfaces which wait for STP delay to be over until they are really forwarding packets. VLAN interfaces that are dependent on their parent ethernet interface. VPN and other tunnels. So you get stuff like "start wpa_suplicant for 802.1x, take the ethernet interfaces up. start the bridge interface. check if DNS works for the VPN. then start the VPN daemon and the VPN interface. then start network services listening on the VPN interface". That would be a typical setup for a server in a remote office somewhere.

Another fun situation is a server with LVM, RAID and encrypted disks (also typical for a remote office somewhere). SAS disks will come online (after spinup time), generate an event. LVM and raid detection will run, generating more events, usually dependent on one another. Device naming is in some cases dependent on the event order, so you also have to rely on systemd-udev not fucking up by-id or by-name links. Then you have cryptsetup, which decrypts the disk, generating another event, but only upon an automatic or manual all-clear and password from the (hopefully working) network. Then you arrive at the actual filesystems in fstab, after a chain of udev events, each of which (and especially the LVM and cryptsetup ones) sometimes fails. Same "fun" for SAN devices.

Oh, and another fun problem is systemd exhausting randomness for too many UUIDs, so cryptsetup hangs for swap devices because those get a key from /dev/random.

Of course those bugs get often fixed over time, but only until the next RedHat release arrives with yet another systemd-somethingd, networkmanager, systemd-udev-bug that introduces another new problem in the above. Usability has gone up since RH7, but reliability is definitely down.

This is a long list of complexities, but I'm not sure how they relate to udev/systemd. Yes, setting up aggregate interfaces, VPNs and fancy drive configurations is complex and can fail due to various delays. That's not a problem with udev reliability if an underlying system fails. We had the same issues before systemd/udev was a thing.

> cryptsetup hangs for swap devices because those get a key from /dev/random

You can choose urandom (third column of crypttab). Systemd doesn't enforce anything here.

You wanted to know how many udev events and therefore dynamic behaviour a server could have at boot. I told you my experience.

And of course the underlying system may fail, but actually it is more common to have udev, lvm and systemd fail. Just google cryptsetup, udev, boot, lvm, raid and problem in arbitrary combinations. Most of what you will find are bugs in udev, lvm or systemd.

urandom is a workaround for stupid behaviour from systemd. Of course /dev/random isn't necessary, but that is beside the point, systemd again broke something by being boneheaded.

Being able to hotswap more components in the data center without power cycling the server is becoming more useful again (I say again because this was a long standing benefit of mainframes). More than just hard disks, it's useful for SSDs, and now GPUs and other types of accelerators. PCIe supports hot plug, but it's rarely actually used.

Not just mainframes, there was and is even RAM and CPU hotplug x86 hardware (but rare and expensive). Sun hardware (enterpricey, but not mainframe per se) had CPU boards you could pull and replace while running. Network cards and HBAs are other more typical examples of things you could hotswap. But the trend has favored software based solutions providing redundancy on multiple cheaper servers. Imho rightly so.

> If it has booted once, it will always boot again, in the same order, reliably

Or not if one of them fails suddenly. That’s just a false sense of stability (not saying it could not be stable, but that is not due to having a sequential startup)

Services should have proper dependency chains and that’s not that hard of a problem to overcome. Without them every other part of service management will be incorrect.