Systemd, 10 years later: a historical and technical retrospective (2020)
blog.darknedgy.net
blog.darknedgy.net
> Perhaps as BPF subsumes Linux into making it a managed-runtime hybrid kernel with subsystems becoming increasingly componentized and instrumented (with early signs in the proposal to add Linux Security Module hooks to BPF programs), as pidfd/process descriptors allow for reliable supervision to be distributed across self-contained processes, as the native Linux mount API becomes more event-driven on its own, and as a generation of Rust fanatics in their undying religious zeal insist that all men are obligated to offer sacrifices to the borrow checker, a new shift may emerge where once again, the init system is made to cease mattering.
Alpine, Devuan, Gentoo and other devs have kept init choice available.
Nothing stopped anyone, including yourself, from modifying or forking a distro like Fedora to make it support non-systemd, it's just you can't expect others to do it for you.
Others made this inconvenient.
It's not about expectations it's about distributions going out of their way to remove choices from users in some highly opinionated ways.
Which is all fine, I just won't use your distribution, what surprises me are the people who feel the need to step in an "defend" this rather pointless outcome for some reason. My favorites are those who suggest that this had to occur in order for linux to "mature" or "become modern."
Pottering et al went out of their way to make this harder than it would otherwise be (e.g. adding the gnome hard-dependency on logind which they then variously claimed was necessary, was unintended, was accidental...)
Nothing's stopping me from modifying Windows to do what I want either, it's just hard. There was a time when Linux people cared about whether their systems were easy to modify.
e.g. GNOME Shell now uses systemd user services to manage its sub services rather than starting them itself. Why? Because systemd does service management a lot better and provides proven features (logging, sandboxing, cgroups etc) rather than GNOME having to re-implement them.
On the contrary, as the article describes, Pottering and the RedHat team pushed all these projects to adopt their stuff, and part of that was making the patches themselves and saying a lot of things that turned out not to be true. It absolutely wasn't an organic success on its technical merits.
Forked by Oracle, Rocky Linux and AlmaLinux, for reasons other than systemd.
apt-get install sysvinit-core
(And more importantly it killed all other well-behaved cooperative init options - this isn't about systemd vs sysvinit, it's about systemd vs standards-based init systems that play nice)
EDIT: this is mentioned in the article, thanks koalacola
Tacitly: most people don’t like launchd, but they don’t interact with it either. Find me a person who writes or debugs launchd plists and you’ll find that:
1. they’re quite rare
2. they don’t like launchd
launchd and systemd aren’t even comparable, one is part of a monolithic proprietary operating system and the other is an ecosystem that is semi-modular which lends itself strongly to being an entire layer of your computing experience (dubbed: “the system layer” by Lennart Poettering).
There is an element of “do it yourself” with Linux, which leads to people (especially people with entrenched knowledge) being wary of things that cannot be debugged easily- but launchd exists in an ecosystem where “take it to the store” is a legitimate fallback, and even if it weren’t there is a built in backup/restore system that literally restores the machine to its entire previous state.
Disregarding the proprietary or “creep”/complex discussion about systemd/launchd:
launchd is an init.
systemd is an ecosystem.
> [...]
> 2. they don’t like launchd
This is true in my case, although the main thing on my mind when I work with launchd my main complaint often boils down to 'I wish this were more like systemd'. IME, systemd has much nicer CLIs, more capabilities, clearer and more comprehensive documentation, more abundant examples, and more conveniences/less privileges required for per-user stuff. The simple INI format for systemd units is also a lot easier and nicer to deal with than the XML format that plists use, even though there's a lot of tooling around plists.
> even if it weren’t there is a built in backup/restore system that literally restores the machine to its entire previous state.
I get this for free on NixOS and it's really easy to get on other distros by plugging /etc/systemd into etckeeper, even without a snapshotting filesystem. (Plus many distros have backups via snapshots of CoW filesystems built in just like macOS does.)
> launchd and systemd aren’t even comparable, one is part of a monolithic proprietary operating system and the other is an ecosystem that is semi-modular which lends itself strongly to being an entire layer of your computing experience (dubbed: “the system layer” by Lennart Poettering).
Benno says literally all of this in his talk. He even has a diagram with the phrase 'system layer' up on a slide while he's talking about it. I just re-watched it yesterday. In case it's missing from the linux.conf.au version, which is the one linked to in the LWN article linked to by the OP, here's the earlier BSDCan version, which is what I rewatched yesterday: https://www.youtube.com/watch?v=6AeWu1fZ7bY
I recall the initial systemd stuff that was constantly posted etc. but now I routinely write unit files and it's really nice. I'm sure the constraints that systemd introduced were a pain in the ass for maintainers but for a user it's been tremendous. Admittedly, some times I just a service unit instead of a mount unit for FUSE and so on, and I still have trouble with `After` and `Before` and `Wants` but overall I rather like it.
Now I have to go check what happened about PulseAudio which is the other Poettering work that everyone and their uncle hated (or maybe it was just Ulrich Drepper who hated systemd). It's funny to me that in the end, my teenage self was some sort of equivalent of a k-pop stan: applying Con Kolivas's scheduler patches, and choosing Poettering over Drepper or whatever. And now all that memory has faded but the names.
Nah, it stayed flaky well past then.
> it was absolutely an improvement over the previous system which either needed manual configuration per program to choose whether to use ALSA, OSS, EsoundD, or Arts with success of portaudio and other sound systems too keep things interesting.
No it wasn't. It was just one more annoying option in the list.
> Even if you did get things working, I distinctly recall having to restart Firefox in order to play music because Firefox had the only available connection to the sound device.
That was a misconfiguration that was possible to do in some systems, sure. There were plenty that avoided it.
I guess modern apps were developed against those APIs that they've become a necessity now?
When I used to use it, it worked nicely, but I started using pulseaudio around the time of its most recent release, when I got frustrated by having to tweak asoundrc to use a USB microphone, and being unable to figure out how to bump up its sample rate.
Then why was PipeWire introduced as a PulseAudio replacement, and why have so many people sung its praises?
I think more people continued to have issues with Pulse for far longer than you realize.
> Then why was PipeWire introduced as a PulseAudio replacement
One of them is putting more focus on permission management with support for sandboxed containers, which is kinda necessary with the direction the Linux desktop is going. But it also is just a backend, that supports jack, alsa and pulse interfaces, so it was mostly a drop in replacement for the applications.
Talk with the guy behind pipewire https://www.youtube.com/watch?v=_buKw1la_tg (4:34 for the motivation behind pipewire)
Systemd can do that for you, too!
How else could systemd purge left over processes (e.g. GNOME tracker and gvfsd)?!? You don't expect developers of the inevitable perfect Linux desktop to fix their code? No lets break the system and take down the well behaving applications with the fallout unless the operator knows (and has permission) to restore the old behaviour. Oh and I hope you didn't expect screen/tmux/dtach/nohup to work.
loginctl enable-linger <user>
If you're using systemd as a service manager (which you probably should do, if your hosts are running systemd?), you don't need the user running your services to be a system user. You can just have (either) your services launch from a systemwide systemd service, or make sure your user has lingering enabled if you're using systemd user services.
If you're not using systemd to manage your application's services, you can still use the methods above if you just have systemd launch your process supervisor of choice (which you should definitely do if your hosts are running systemd and you're using some other process supervisor).
If you want to leave user processes running when you've started them manually and no user session exists anymore, you can unset KillUserProcesses in logind.conf: https://www.man7.org/linux/man-pages/man5/logind.conf.5.html
I don't get all the hate, that primarily argues about the philosophy of Linux, when the philosophy made everything just harder..but I guess some people just want to suffer for their believes, mich like some Catholics
It was about the udev/systemd maintainers writing "fuck you Gentoo developers" in a mailing list when they wanted udev to support only systemd, actually that udev should be part of systemd (for no real reason) and reimplementing things (sometimes badly for years) instead of reusing what was in place. It was about pretty big bugs (I remember a bug around coredumps being cut at ~700MB in size, which the systemd maintainers responded with the equivalend "well that's your problem not ours, journald is not fast enough and so we cut the max size" and was only closed eventually - ~3 years - probably because some RH customer screamed loudly enough). And the original discussions were even before systemd decided to hardcode what essentially is random IP addresses into resolved ("maintainers should change those values if they don't like them!") and other network services (NTP? I don't remember now anymore). It was about the billion libraries (later on there have been some security fun issues) it brought in and made your stuff link to (I vaguely remember something to parse qr codes?). It was about not-documented DBUS interfaces and making DBUS a core part of the system...
Most complex services still had heavy, long prestart script written in bash anyway at that point in time even when using systemd, so the script complexity was not really gone (and I think postgresql for example still has something like that).
Now, Redhat pushed hard for it and you cannot just ignore what they do and go your own way since the number of maintainers they employ is massive along the entire stack, from kernel to GNOME - Systemd was convenient to use and people went for it even with all the issues it had/has for good reasons (same as pulseaudio), but thinking the only discussion at the time was about the "linux philosophy" is a strawman and changing history a fair bit. :)
"But it is just harder to learn how to write proper software that would work on a machine slightly different than my own"...
I've seen some software depend on libsystemd for the sd_notify but I think after the whole security kerfuffle there won't be many of those anymore.
If you just wanted simple/easy init scripts you could do that with runit for years without having to switch dozens of unrelated system components over to the systemd-approved version and render your system unbootable every so often.
I like it a lot and my team uses it extensively to precisely specify the behavior of our embedded products. systemd has a low memory footprint for what it does. It also helps us make our products more secure by making it easier to isolate processes / different system services as much as possible. So the risk of getting hacked over a network service is reduced a lot. Without systemd we would be forced to create our own which wouldn't be as secure.
The service manager? It manages services with many of the same features of systemd units, including advanced features, like firewall integration. This is combined with windows event system which provides many of the same features of the systemd journal.
> making it easier to isolate processes / different system services as much as possible
These features are built into the operating system. systemd provides a particular method of accessing them, but certainly not the only one.
> So the risk of getting hacked over a network service is reduced a lot.
Your risk of getting hacked has nothing to do with systemd. You are hoping that the easy process isolation features you're referencing actually work and thus _mitigate_ the potential damage of that hack. This is not bad strategy, but is on it's own, entirely incomplete.
> Without systemd we would be forced to create our own which wouldn't be as secure.
At the risk of being hyper critical, this is precisely the type of logic I would use to sell snake oil, too.
Some DBus message is not appearing and it stalls out your boot process, finding where it was supposed to come from and why it didn't appear is a nightmare. If you want to use a different daemon than the one SystemD provides for some service it will fight you every step of the way, sometimes even undoing whatever you had to do to make the process work in a random update. As someone who experiments with weird networking stuff from time to time I've been kicked in the nuts by NetworkManager on several occasions.
And what does this mean of the amateur radio operators who call themselves HAMs?
nit: NetworkManager is not the one provided by systemd in the first place - that's systemd-networkd.
That aside I do wierd networking stuff for a living and here's how I see the available tooling:
* NetworkManager - great for a laptop where you're moving between networks and don't need anything too fancy.
* systemd-networkd - great for servers and other mostly static networking setups, even weird ones.
* If you're doing anything weird dynamically (e.g. experimenting) - disable the above or move the interfaces you are experimenting on to a fresh netns and do everything by hand with the ip command (and maybe udchpc or dhclient if you need a dhcp client, but be aware of how to write your own callback scripts for how to handle various associated events).
I'm not entirely sure what you mean by this, but linux (the kernel) is very much not a "do one thing and do it well" kind of program. You'd need a microkernel for that.
With modern software it's a pain in the butt. Remember CD burning software? That at least used to invoke cdrecord underneath, and whenever that went wrong it was awful, because text is an awful interface for a complex system.
Today you'll find many things that actually "de-UNIX-philosophy" stuff, by making a library that wraps around an invocation to a complex tool and presents what 99% of users actually want -- a library style interface.
>Today you'll find many things that actually "de-UNIX-philosophy" stuff, by making a library that wraps around an invocation to a complex tool and presents what 99% of users actually want -- a library style interface.
There are tradeoffs to this simplicity. Your "library" is most likely a pain in the ass to use even for people using the same language, much less other languages. The output and invocation of a simple program is less likely to add a breaking change than its code. Do you think libraries were just invented, or what?
I like the idea of making the library first and then using that to build a tool. But that may not make sense in all cases, and it's extra work as well. For people using other languages, or trying to write simple shell scripts, a library alone is not going to work.
(2020)
Discussion then: https://news.ycombinator.com/item?id=23204666
They screwed up the libsystemd first by bringing tons of dependencies (including libxz), and then inventing a braindead delayed loading mechanism. That required adding custom ELF sections. It's funny that they used "macOS does this!" as an argument for the delayed loading, when in reality macOS removed that functionality about a decade ago (for being braindead).
While all they needed to do was simply split libsystemd into two libraries.
What are you talking about???
dlopen is hardly a new "delayed loading mechanism" and having a custom ELF section where the dlopen'd libraries are listed is just a positive thing for package managers, so that it can be automatically detected during build just like used SO files (for writing the build metadata files).
This: https://github.com/systemd/systemd/blob/main/docs/ELF_DLOPEN...
> dlopen is hardly a new "delayed loading mechanism"
Yes, it's a well-known anti-pattern that libraries like glibc are trying to minimize.
When your fuckup requires a change in the basic system architecture, you are probably doing it wrong.
_All_ they needed to do was split libsystemd into libcoresystemd and libjournald. Only the latter requires libxz and other compressors.