Systemd, 10 years later (2020)
blog.darknedgy.net
blog.darknedgy.net
There is strange false history that systemd replaced sysvinit, which occurred only on Debian, because Debian was the last major system to discard sysvinit.
Gentoo had replaced sysvinit with OpenRC in 2006; Fedora and Ubuntu were using Upstart in 2005. systemd primarily replaced Upstart and systems that used Runit or OpenRC tended to have stayed with them.
Frankness be, this false history seems to be crafted in order to compare it with very outdated technology to make it look good. Obviously systemd is better than sysvinit, technology that almost all systems but Debian had discarded long before systemd was even started. The issue is whether it compares favorably to Upstart, runit, and OpenRC, and to the first I would say yes, because Upstart was full of very ugly hacks and didn't deliver it's ambition. In the second two cases, their mode of operation is so different that it's hard to compare. runit was inspired by daemontools and is very simple and effective and works in a very different way from most models of service management; OpenRC is backwards compatible with sysvinit and is essentially a large library so that init scripts typically need nothing more than something such as:
#!/sbin/openrc-run
command=sshd
args="-D"
daemon=no
provides=ssh
after=net
The previous gentoo init system was already better than sysvinit and initscripts.
I still prefer and use systemd though.
And why wouldn't you compare it with the systems it actually competes against and replaces rather than against something that was barely used any more before systemd even stated development? The comparison with sysvinit, suggesting that those that do not use systemd advocate sysvinit instead, is simply a flagrant straw-man designed to make systemd look good.
I would say that's a meaningless comparison and those other systems were never considered as real replacements. At that time I don't remember OpenRC getting any traction outside of Gentoo, and the distros using Upstart were already unhappy with it. Adding to that, systemd explicitly was targeting distros that used a lot of sysvinit scripts and was intended to be used as an upgrade for them. It's largely succeeded at that.
So the year you're looking for isn't when OpenRC was introduced, but rather when Gentoo started using the OpenRC style of init script, which happened much earlier - it's definitely been like that since I started using Gentoo, at least since 2003 or so.
It's actually a misconception that sysvinit is the whole init system; sysvinit is PID 1. Gentoo still uses sysvinit for PID 1 by default. What matters is what your service manager is - that's OpenRC here.
But on the desktop I use FreeBSD which has a different system again.
By the way alpine is also working on a full service manager init system (like systemd) but more minimalistic, based on s6 service manager. Looking forward to seeing how that pans out.
https://skarnet.com/projects/service-manager.html
When something comes from the alpine team I have a much higher confidence that I'll like it than from RedHat :) I don't care for Gnome either as it has become so opinionated.
I would add S6 to the list, as this would be the better challenger to systemd. That said, I use OpenRC daily on Alpine servers with little to complain about.
This is partially incorrect and partially misleading.
Incorrect in that openSUSE and Arch are major distros and migrated from sysvinit to systemd.
https://news.opensuse.org/2011/12/22/systemd-e2-80-93-boot-f...
https://www.reddit.com/r/archlinux/comments/4lzxs3/why_did_a...
Misleading in that Ubuntu and Fedora did use Upstart but not in its native mode, they used it in its sysvinit compatibility mode. This is almost identical to sysvinit, so from the point of view of a user or a package maintainer it was functionally barely distinguishable from sysvinit.
OpenRC was considered by Debian, but 7 out of 8 members of the technical committee preferred both systemd and Upstart to OpenRC (Ian Jackson preferred sysvinit over OpenRC and "further discussion" over every other option).
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=727708;msg...
OpenSUSE also had upstart in the meanwhile but continued to support sysvinit as an alternative.
Arch Linux had sysvinit as it's pid1, but it's big sell initially was that, similar to Crux, it had a custom BSD-style init script setup and did not use sysvinit's booting and rc mechanism.
OpenRC systems typically also continue to use sysvinit's pid1 to this day, which is not the complaint people have with it but the impossible to understand script sand lack of dependency management of it's service management because it was never meant to do service management which was a bit of a hack.
> Misleading in that Ubuntu and Fedora did use Upstart but not in its native mode, they used it in its sysvinit compatibility mode. This is almost identical to sysvinit, so from the point of view of a user or a package maintainer it was functionally barely distinguishable from sysvinit.
There is no such compatibility mode as far as I know. A big sell of Upstart and OpenRC was always that it was backwards compatible with old-style sysvinit-style scripts even though new ones were written in the new style. There is no global mode as far as I know but no doubt many of the olds scripts remained.
> OpenRC was considered by Debian, but 7 out of 8 members of the technical committee preferred both systemd and Upstart to OpenRC (Ian Jackson preferred sysvinit over OpenRC and "further discussion" over every other option).
That would not surprise me. It's not what Debian is looking for.
I wasn't aware of that ... it turns out that it was the other way around: openSUSE 11.3 has this item, implying it's not the default:
"Upstart is included as optional init system"
https://en.opensuse.org/Archive:Product_highlights_11.3
openSUSE 11.4 then mentions systemd, although "experimental".
https://en.opensuse.org/Archive:Product_highlights_11.4
openSUSE 12.1 defaults to systemd (see previous comment), with sysvinit as non-default option.
So Upstart was a short-lived non-default option.
> A big sell of Upstart and OpenRC was always that it was backwards compatible with old-style sysvinit-style scripts even though new ones were written in the new style.
You are right that there are no distinct modes as such, my memory is quite fuzzy as it's been some years, and I wouldn't rely on anything I say. And it looks like Ubuntu (unlike Fedora) did replace a lot of init.d scripts with native Upstart jobs.
https://launchpad.net/ubuntu/+spec/replacement-initscripts
I can't remember what the problem was with mixing init.d scripts and native Upstart jobs, or what impact it had, maybe it was not important in practice.
The Debian wiki has this item, but, well, it's a Wiki, so perhaps it's not the most reliable source.
Can't set proper dependencies until everything has converted from SysV init files to Upstart jobs.
At 82k loc it's huge for what it does, but nonetheless is order(s?) of magnitude smaller than systemd. If it was active I'd trust it more that systemd on that basis alone.
I can't say I'm a fan of systemd's event driven approach. A system whose state depends on it's configuration plus past state is usually a nightmare to debug and control compared to a system who state is dependent on it's configuration alone - and that's how it has worked out for me and systemd.
The simple cases work fine of course but systemd's power over what came before it is it's dependency relationships it allows you setup. But they are the very things that break in weird and wonderful ways when the events it's reacting to fire in an unexpected order. If you could set them up reliably without much thinking it would be a big win - but as it is, not so much.
The vast, and I mean vast majority of distros used SysV before transitioning to SystemD
Also upstart, which was actually pretty good, but systemd was more ambitious.
Additionally Upstart was not good. It has some good ideas but its approach to dependencies was generally regarded as unintuitive. Writing services in systemd is much more obvious than writing for Upstart.
If you spend an afternoon installing it and configuring a few network services, you’ll already know most of what there is to know about how init used to work.
It used to be extremely elegant and robust.
The main flaw in the design was that the “service” files were written in a Turing complete language, which meant that distribution maintainers could ship things that were arbitrarily terrible, and many chose to do so.
(Edit: Also, the organizations that somehow couldn’t figure out how to use and maintain traditional init systems were the ones that produced the current generation replacements and pushed them so hard.)
This is revisionist. The organizations that built the current generation were those who had to support those who couldn't figure out how to use and maintain "traditional" init systems. Distro init scripts were (usually, and at great effort) decent, whether from Red Hat, SuSE, Debian, etc. (just don't ask too hard about why they were different). But the third-party (often, but not exclusively, proprietary) software vendors looking to support even a single distro were godawful. Eventually the support burden for e.g. Red Hat got large enough it made more sense to replace it wholesale than keep mentoring the latest batch of e.g. Oracle interns in how to write half-decent shell scripts.
yes, it's still open source, but modifications and inspection are far more complicated creating a much higher barrier to entry.
in the earlier days when it was still buggy, even linus himself was screaming about not being able to easily take the case off of pid 1 when it misbehaved.
I’m waiting for Microsoft to buy Red Hat. Failing that, IBM.
I doubt they will though. Like others have said MS buying Canonical would make a lot more sense. And there have already been a lot of overtures.
The UNIX philosophy isn't meant to be taken as gospel - hell, it's very hard to define what "doing one thing" even means.
Yes.
Emacs development began during the 1970s at the MIT AI Lab, whose PDP-6 and PDP-10 computers used the Incompatible Timesharing System (ITS) operating system that featured a default line editor known as Tape Editor and Corrector (TECO)
GNU Emacs later (1984) was based on Gosling Emacs (1981), which was written in C and extended with Mocklisp. Mocklisp was a rather uncomplete Lisp, which was replaced with Emacs Lisp by Richard Stallman.
I use MacOS and iOS for internet viewing and A/V. I have no interest in how they are implemented - superficial observation indicates they are a mess. I do ssh into my servers from MacOS - it's comfortable with my laptop actually in my lap.
There were several init replacements in the mainstream before systemd. OSX and Solaris both had their own, and they were as UNIX (officially certified) as you can get.
Also, as long as we're doing non-sequitors; Microsoft is more likely to buy Canonical than they are to buy RedHat or IBM.
The piece of software systemd is most like is launchctl, which doesn't come from Windows.
On the other hand, the biggest "Windows-ism" to reach Linux lately is io_uring. But it's both too technical and too useful to have some populist uprising against it.
I don't care about the init system - I reboot a Linux system once in a blue moon, for kernel updates. It's the rest of the complication and changes to a system that I had already made the effort to master. I understand that the New Linux is better for Enterprise, so that's what is going to happen. That doesn't mean I have to like Poettering gradually re-writing the entire base of userland. I just want to run simple servers, not a major enterprise.
The biggest problem I had was that it was difficult to keep upgraded. In those days, it was basically a re-install every six months, plus a merge of any local changes. I understand things are much better now, and I may give it another try.
I would argue that locked and pentalobe metaphor is not very accurate - systemd is no more locked than most other open source software.
Personally, I was more of a fan of upstart then systemd, but both were a massive improvement over sysvinit in terms of offering an actual interface with abstractions for the things you wanted to do. This is again from the perspective of a user, but that's where (in my opinion) most of the cost/benefit was anyway.
A monorepo containing well-organized boring C, developed on github of all places, is the polar opposite of raising the barrier to contributing vs. what was replaced.
Full-disclosure: I contribute to systemd.
But, like some other comments have said, I was young enough to dodge the drama. I basically grew up with systemd and I don't feel like I'm missing anything. The biggest problem I have is that Ubuntu Server defaults (or used to default) to having the system wait on network interfaces, which is annoying when I want to build a desktop off of it. And sometimes on shutdown it waits for some daemon I don't care about to quit. Which is probably just a bug in the daemon.
But now that things aren't buggy, being unable to modify them in place is a good thing. It sets a clear boundary that this is not how you do things on a modern system, so you can always expect systemd to be the same, consistent, non customized thing it always is.
i'm not going to argue it's a bad thing, i actually really like macs and think they hide a lot of details making time and space for other pursuits, although since you cede user serviceability it's important to purchase a service plan and that you don't try anything that wasn't provided for in the design.
maybe simple composable parts are past their prime. i always loved the simplicity of inetd. it was both self describing and inclusive. you could write an internet service without any knowledge at all of sockets and learn about the system by simply looking at it. but i suppose that's given way to large monolithic daemons these days.
personally though, i always thought the simplicity and the composability of unix philosophy was what bred systems that outperformed their competition in terms of security and extensibility. so many commercial systems reflected apparent power grabs within the corporate structures that produced them... i suppose that when you see foss monoliths like this, it's the same game playing out on a different field.
You could achieve the same thing with systemd socket activation :)
neither. last i checked, linux, even with rt_preempt, was unsuitable for realtime control applications.
even hollywood gets that, didn't you see the end of "don't look up"?
To me, the most poisonous example that contrasts the traditional design with Freedesktop's design is acpid contrasting logind. In the former case, an acpi event is sent, and acpid simply executes a file such as `/etc/acpi/action/lid_down.sh`. That file can contain anything the system administrator wills, from a simple script that does nothing more than `systemctl sleep` to whatever complex task he could desire.
The last time I checked the situation, logind had replaced this powerful, and simple mechanism with a configuration that offers nine options to execute when the lid is pressed, none of which involve executing a simple file. Of course I can edit the source code and recompile to allow it to do what I want it to, but I would far rather simply quickly alter a simple shell script to do so.
And luckily I can, because I do not use logind but acpid, but that does not change the ire I experience when dealing with many Redhat and Freedesktop people who make it very clear that I should be using logind, that it's better, and that they create artificial dependencies on it to as Poettering calls it “gently puh” me in that direction.
But it does not change at all why I dislike it, and why I am annoyed by Freedesktop's pushing and insistence that many keep to these more custoizable solutions supposedly simply because they are “stuck in their ways” and that their philosophy is the future and “modern”, whereas all I see is more hurdles that stop me from effectively being the master of my own machine.
"Customizability" is also not the domain of a group that like Freedesktop that publishes specifications. Their purpose is to define some baseline of standard behavior for interoperability. The customization comes from individual projects. No offense but your criticism is pretty misplaced.
They aren't particularly keen on hacking in how they design their software and standards and talking with them is frustrating to say the least.
If Red Hat engineers submit something there it's because they think it's useful to the other desktop projects. It's just a collaborative space to put specs. If you have a spec of your own you can just create a new RFC. You don't need to persuade them to change how they design theirs, unless you intend to start working at Red Hat and start maintaining some of their software for them.
The difference is that everyone outside of Red Hat an Freedesktop isn't creating artificial dependencies on each other's projects to “gently push them” so people outside of it again have to write more code to decouple them again.
They did write the code to fork elongind in such a way that it no longer required systemd; they did write the code so that GNOME could run without logind like every other system; they did write the code to fork of udev from systemd again so that it could run without glibc. — None of these projects suffered loss of functionality.
They're complaining that they have to, because every time, these dependencies that have no technical justification happen, it's always one RedHat project depending on another to create product tying and encouraging adoption.
The rule of capitalism is simply that product tying works in practice, however bad it is for the user and consumer. That's why Apple likes to come with custom cables that only work on their hardware for no technical reason, and that's why logind was incorporated into systemd and came to depend on it for no technical reason.
It sounds to me like the bazaar system is working fine qua its own standards, then. Red Hat writes what they need to solve their problems, party X writes what they need to solve their problems, the code is shared, everyone gets what they want.
Except: It sounds like what you really want is for Red Hat to solve your problems, not only their problems. That's definitely not how this stuff shook out historically. And it opens a door you might not want to open, because now you need to ask yourself why users liked systemd, and what obligations you have to support beyond your own needs in your own software.
The difference is that Red Hat's problem is not making enough money, and everyone's problem is software not working because Red Hat breaks functionality to make more money.
Red Hat's “problems” are similar to the “problem” Apple experienced that that heir hardware was interoperable with third party hardware, so they made sure to use proprietary cables that only work with Apple hardware.
> Except: It sounds like what you really want is for Red Hat to solve your problems, not only their problems. That's definitely not how this stuff shook out historically. And it opens a door you might not want to open, because now you need to ask yourself why users liked systemd, and what obligations you have to support beyond your own needs in your own software.
No, what I want is for Red Hat to stop product tying, which you conveniently ignored and didn't address.
What they're doing isn't solving any technical problems any more than Apple is doing by using proprietary cables when standard cables suffice.
I didn't say it did! I said if you really buy into the "Linux is about choice", "Red Hat stole our fun OS" etc. bullshit that always circumscribes the populist anti-systemd position in these debates - then you don't really have a moral or tactical position other than "shut up and code."
I don't think that's true - I think there are other obligations to users which are just as important - but these all suggest systemd even more than the "master of my own machine" nonsense.
I don't believe that. People commenting negatively on some Linux software/trend do not need any such privilege or permission or show of stake. I think anybody with Linux experience is entitled to his/her opinion on what should and should not be done on their Linux machine and in the Linux software space, whether they produce new code or not.
I don't use systemd on my systems, I prefer sysvinit and openrc, and I sometimes express my arguments for why. Why would I have to write my own init system just to express my opinion that for some usecases, other software is better than systemd?
How so? The canonical solution would be to ship a default `lid_down` script that could take the currently-active policy as set by the GUI as an input. But the nice thing is that adding/tweaking policies is a fully-supported operation, you just edit the script and the GUI config mechanism to support them.
I think I figured out my issue with systemd. It does not apply Chesterton Fence when redeveloping things… and how could it, when it redevelops so much.
Maybe it’s fine if it’s all you’ve ever know (or you never hacked on your system) but those that did have lost something.
They are simply annoyed by A) it being pushed on them; B) people often acting that they are irrational for not liking it while it does not what they expect and need; and C) discussions with Freedesktop developers that, frankness be, reveal they live inside of a very strange bubble and have no idea of the use cases outside of it.
This is software brought to you by the minds who brought you. “I have no idea what XFCE is or does, sorry.” or that libinput did not need a way to disable mouse acceleration because everyone wants mouse acceleration.
Your issues with XFCE and libinput don't seem to be related to systemd. I can't see how airing grievances against other unrelated projects would be a constructive place to take this discussion. Complaining about open source projects only supporting some use cases is also confusing to me. A lot of these projects are very open about the fact that they exist only to "scratch their own itch".
I have a problem with Jon McCann's famous quote, who was at the time a lead developed of GTK+, who did not know what XFCE, one of the biggest consumers of the library, even was when he made a change that broke about anything outside of GNOME. — What his language suggested was that he lived in a bubble thinking that only KDE and GNOME existed, and that since KDE wasn't using GTK+ it was fine to do this.
Poettering has come with very similar claims that betray that he does not understand what people outside of his small circle want and need, the same can be said for Wayland developers, DBus, NetworkManager, PulseAudio and all such other infamous projects.
QT developers will not tell you “I have no idea what LXQt is or does.”; they do not generally break things that compromise 90% of their consumer base and they do not remove theming because they fear it's existence will hurt the “brand identity” of KDE.
It's dishonest to say similar problems occur outside of the Red Hat circle and that circle alone is commonly criticized on these policies.
If you're trying to prove a point, you could at least mention something that actually caused a real problem instead of taking this one out of context. I have no idea why anyone would keep mentioning this quote or find it memorable, it seems completely insignificant to me. You also seem to be ignoring all the positive interactions this engineer (or any other Red Hat engineer) might have had. I can personally name a lot of instances where Red Hat engineers have fixed upstream bugs that were affecting me. And just to make it clear I'm not saying this to single them out for praise, a lot of other companies fix upstream bugs too. That's how open source is supposed to work.
And you actually could go and search around on old mailing lists to find similarly questionable 11-year-old quotes from Qt developers if you really were interested in digging up more old drama. These things happen everywhere that people go because people don't agree on everything. I suspect you also think that's ultimately futile though, so why keep flogging this particular dead horse? This is still way, way outside the scope of discussion for systemd anyway.
Or what, is everyone except you under thrall to Lennart Poettering, master wizard?
I have very specifically only criticized Red Hat developers. I have no problem with any of the others and they don't show the same problems.
But that raises the real question: If the problem is systemd, and the problem is so bad, why do you only blame Red Hat and not the every other distribution that switched to it? Do you think they were all victims of Tricksy Lennart, or?
And you also brought in SuSE, Arch, and KDE, which I never criticized to make your argument that I simply hate everything work, which is quite disingenuous.
> But that raises the real question: If the problem is systemd, and the problem is so bad, why do you only blame Red Hat and not the every other distribution that switched to it? Do you think they were all victims of Tricksy Lennart, or?
I do not, and have never blamed distributions for switching to systemd.
They can use what they want and it's neither their fault nor problem that systemd and other Red Hat projects are known to create dependencies upon one another for ill technical merit.
I have criticized Red Hat projects for creating dependencies on other Red Hat projects for political, rather than technical reasons; the rest is simply your putting words into my mouth to make a straw-man argument function.
https://trac.transmissionbt.com/ticket/3685
The ticket is about GNOME 3 removing support for status icons from the GNOME panel. GTK3 still has the status icon API, although they're deprecated and don't work on Wayland. That shouldn't have any effect on XFCE or any other shell with an old panel that supports the old X11 status icons. I think you can also restore the functionality to GNOME with an extension.
Also, if you check further down the issue you'll see this clarification:
>There should be no change in behavior for non-GNOME platforms.
You mean you want a flat acceleration curve? You want to accelerate the mouse, otherwise you need a gaming mouse with the DPI set very high and then keep going into the mouse firmware to adjust the DPI whenever you play a game or switch between different (scaled) resolutions. Libinput has different acceleration profile defaults for different devices, but you can just change it to the flat profile which I prefer as well.
See https://wayland.freedesktop.org/libinput/doc/latest/pointer-...
Basically the proposition is akin to a weapon exchange program: hand over your powerful but dangerous Turing complete configuration language in exchange for nicer looking desktops.
Who would not prefer a better looking desktop?
And if it simply remained at that, but left me alone I wouldn't be so irate, but I don't feel left alone and “Who would not prefer a better looking desktop?” already borders on this “My way, of the highway.” philosophy of Freedesktop, whose developers all too often tell one that one's subjective likings are wrong and that one is somehow “wrong” or “outdated” for favoring a machine that is fast to configure and does what one wants.
It also doesn't help much that on a purely visual level there design choices look ugly to me, and that these are also the same people that remove theme options lest the holy “brand identity” be compromised, as well as that their design blasts too much bright white light into my face when I'm already tiring my eyes out trying to learn to read a language that's not written in a script I'm accustomed to.
-- Sent from my terminal
I face the very same issue in my day job where I'm struggling to cut the right balance between a featureful data manipulation language with which the customer can implement whatever he needs, as long as he is OK with a bit of programming, or a more limited one for which we have a GUI... and that GUI itself has to be tested, and I'm also currently reviewing QA tools to help testing GUIs, and they basically fall in two camps: those that require one to spend a week learning how to program them, and those that are much more limited but come with a straightforward GUIs...
Before systemd the same conflict happened with network-manager: my network setups were usually custom and out of reach from network-manager that I despised and purged of my desktops first thing after a fresh install. Yet with time network-manager supported more and more use cases and I'm now happy to use it. Maybe one day systemd will also make it possible to add the bits of hackery here and there and we will all be satisfied with it.
For now, if so many users have it easier thanks to systemd, which seems to indeed be the case, then it's still something.
Indeed there isn't. And people that want simpler declarative configuration files can have them as far as I'm concerned.
My ire is the mentality common of Red Hat employees that I am wrong for wanting traditional Unix-style flexible configuration and their dependency politics to try to get others to switch over. If they just designed their software how they want to and otherwise let it be, such as KDE and XFCE are doing, then they would not be drawing the same criticism, but they have a tendency of coming with a dismissive attitude to everyone who wants it different than they do and insist their way is the future and when people point out that their way does not offer functionality they need either for convenience, or their livelihood they dismiss it.
> For now, if so many users have it easier thanks to systemd, which seems to indeed be the case, then it's still something.
If various Red Hat projects didn't find a way to create dependencies on systemd to encourage it's adoption and systemd was simply there for the people that wanted it but left those that didn't alone, then it wouldn't be so controversial.
What I quoted was “Who wouldn't want [...]”; that's the Red Hat mentality, that everyone who has subjective ideals than they either does not exist, or is wrong.
In my experience, that is what they're doing. Their software just happens to be adopted by more users because in a lot of cases it is better and does represent the future. You have that advantage when you support a lot of commercial users over a long period of time. Red Hat also has a policy of contributing a lot to upstream so that also adds to why they have success with other projects.
>If various Red Hat projects didn't find a way to create dependencies on systemd to encourage it's adoption
I think this is an exaggeration. I haven't seen any projects that have anything more than a trivial dependency on systemd to enable some optional functionality.
Systemd may indeed be somewhat more complex, but I don’t think that it would employ many accidental complexity— it is complex due to the problem domain. But imo that is a worthwhile thing to spend complexity budget on.
There is absolutely no need for its command names and everything else about it to be so counterintuitive.
Exactly this, yes; and that's why it was pushed the way it was.
I dread the day when systemd or an init system just like it consumes the BSD world.
Do yourselvs a favour and pick Postgres for new projects. If you properly vacuum your tables you'll probably be fine.
Git has the same problem but at least git is excellent enough software in other ways that it makes up for the horrible UX. Systemd is mediocre.
systemd-cgls
> How do I list units
systemctl list-units
Simply use tab completion (like in zsh). And do not dare to compare UX-nightmare from git with systemd...
I am considered a Linux expert in my field but the truth is that I really don't know how systemd internals work well enough to read that review and fully understand it. The review did teach me some things about what Units really are, more than just services and timers, but other than that it was too deep for me.
And yet I work full time with Linux systems, building services on Linux systems for major clients and government contracts.
From my perspective as a user and operator of Linux, systemd has been nothing but good things.
So far. I hope for your sake it continues to be so.
I actually appreciate, now, what systemd is trying to do, but I am not certain why it has to take over home directories and everything else to do it. And I am really uncertain why it has to have such bad taste. I can get why it uses C, even though IMHO that is a mistake. But .ini files? D-Bus? XML‽‽‽
Oh well, at least it's not YAML.
A normal user will never really link the features they see or do not see back to the init system. I'll draw a parallel with the worst subsystem being cut out of modern Linux - X Windows.
As far as I can tell Linux is missing good macro systems that can record actions taken by a user, then replay them. For loops for non programmers, basically. This isn't because the idea is radical, but because implementing the vision historically needed a programmer to understand X - which is too hard. There is a hidden link between the subsystem being awful and programs not existing.
Similarly, the monoculture created by systemd is influencing how the future of Linux evolves by making some things easier and others harder. End users won't be able to detect those influences though, and unlike X's problems systemd is likely a net positive. Systemd easily improves on the old init scripts.
While I agree with the broader point about (any) monoculture, I don't think that this can be blamed on systemd.
Systemd did not itself create the monoculture. When it came up, it was one of a multiculture. Distro maintainers then chose it en masse so that it became a de facto monoculture.
I'm not saying that necessarily makes it the best choice, though.
On a slightly different note, I find such comments funny in a way, because one of the reasons given for Linux's lack of market share on "the desktop" and, broadly, as a platform for "enterprise" software (think Adobe, etc.) is because the ecosystem is so fragmented. So, essentially, the platform isn't considered because it's not a monoculture (as would be Windows and Mac).
systemd became the de-facto default init sys because Distro Maintainers have chosen it. I posted this link here before, but I will happily post it again: https://www.reddit.com/r/archlinux/comments/4lzxs3/why_did_a...
Such a thing doesn't happen if the thing offered isn't an improvement over what was before.
As a matter of fact, this doesn't even happen if the thing offered is only a small improvement. We have seen this time and time again in Programmig languages: Exciting, clever new things that offer real advantages vanishing into obscurity because while they were better, they were not better by a large enough margin.
Diversity is great if it offers advantages. Different kinds of RDBMS are a good example. Different kinds of shells. Rust being there as a C alternative, Go being there as a Java alternative, Julia as a possible Python alternative, WASM instead of JS. All great.
But keeping something mainstream just because its different, adds no value. People can and do still use other init systems, but I am very glad I no longer have to wrestle 400-line sh init scripts on my production servers.
It does happen sometimes. GNOME3 and GTK3 didn't offer much to Linux desktop users, yet they were adopted.
And improvement is subjective. Maybe systemd made maintainers' lives better. But it also introduced big nasty software full of bugs that discourages hacking by users. Many users consider that a deterioration of the Linux ecosystem.
I'll take wrestling a bug in a 400-line shell script to wrestling a bug in the systemd C code and compiling my fixed version.
Are we thinking of the same thing? GTK3 had a much better theming system, new widgets, support for high DPI displays, improved rendering, support for animations, and a lot of other things. GNOME 3 had a much more streamlined design too. You may not have liked them but there were plenty of new features to offer.
>I'll take wrestling a bug in a 400-line shell script to wrestling a bug in the systemd C code and compiling my fixed version.
I wouldn't, once you fix the bug in the C code it's fixed for everything. The bugs in the shell scripts keep needing to be fixed over and over again, because copying code around in shell scripts is really common. There is also a line to be drawn where low-level code should be written in C and I think this is well past that line. Service script code is usually privileged and runs as root so the utility of making it hackable seems pretty small. I expect you'd agree running bash scripts inside the kernel is a bad idea too.
Fixing a bug in shell script, even if 30 times, is still much easier than downloading, understanding modifying and compiling and deploying 4000 files and 1.6 million lines of systemd C code. There is an obvious cost/benefit factor here.
If systemd fixed just init and basic service management in correct and minimalistic way, maybe fixing its C code here and there by power users would be rational. Not with the ever-expanding pile of prototype-level software we got. I'm not reading that code, not even for money.
For desktop app developers? There were a lot of API refinements and enhancements to the widgets too, I didn't mention them all because there's far too many.
>4000 files and 1.6 million lines of systemd C
Just FYI these figures are incorrect, you're likely including the tests, documentation and data files. Running cloc on the current systemd code base I get around 2100 .c and .h files and 520,000 lines of C. That also includes all the other components too, not just the service manager. The service manager is a small fraction of that, it's only around 150 files including the headers.
>There is an obvious cost/benefit factor here.
From the perspective of a distribution I think it's much more cost effective for them to fix this once in C. To you what looks like just 30 shell scripts can quickly multiply into the thousands when you have to consider this is across all users of that distribution who might all have the same buggy set of 30 shell scripts but with different ad-hoc fixes applied.
>If systemd fixed just init and basic service management in correct and minimalistic way, maybe fixing its C code here and there by power users would be rational.
I think if you actually look at it, you'll find this is already what it does. That's what I saw when I actually took a few hours to familiarize myself with the code.
If a bug happens in this code, everything is affected, everyone screams at the same time, it is obvious there is a bug, and once its fixed, its fixed everywhere and for every service.
If there is a bug in one of the old init scripts, it may go unnoticed for several years, until suddenly someone needs service Bar depend on Service Foo, but Bar cannot figure out the state of Foo correctly, and good luck to whoever has to solve this: unless the combination of services is a popular one, whoever has this problem, is likely the first to encounter this, there is no general outcry because nothing else is affected by it. A fix may not be forthcoming because the behavior may not even be considered a bug (the maintainers of Foo and Bar may simply disagree how some service state is to be communicated), and usually someone has to trawl through a pile of sh scripts (because backward compatible all the way to the 90s it has to be!) and implements a fix, which then gets destroyed by an update when the guy who fixed it is on holiday.
As someone who has been that exact guy, I have to say sorry but no, I don't want to do this any more.
Shell scripts are tunable to the specific system. If a rare bug is in a shell script, I can usually find and fix it, unless the script is insane, in which case I can write a sane one.
If a bug is in systemd, I can't easily fix it myself. I now have to fight upstream to recognize my issue and fix it for me. I don't have a good experience with this. Nowadays I'd rather write a small script to work around that bug if possible. Much faster and much more effective.
That's not true. You can do as you would with the shell script and patch it in your local copy. If this is too much hassle then that shell script workflow shouldn't be broken by systemd and you can always fall back to it. It's still possible to run shell scripts as systemd units. You could also run another service manager as a systemd unit as another workaround.
Also, in my experience, if you're fixing an actual crash, those patches are really likely to get accepted. Upstream appreciates that, I haven't seen them fight anyone over an easily verified issue. If you're submitting giant 10,000 line patches that change the public interfaces and cause regressions, that's a different story.
Because they were forced to choose it due to vendorlock and many other projects became dependent on systemd. Unsystemding everything would be very expensive.
Same problem will be with snap and flatpak.
Snap and flatpak are essentially just package managers, I can't see why you would expect those to be worse than any other package manager.
> Snap and flatpak are essentially just package managers, I can't see why you would expect those to be worse than any other package manager.
Those are even worse than tarball bundles. Main purpose of both overengineered projects is to tie users to vendor's services (snapcraft, flathub, e.t.c.). Centralized delivery is outmoded, future is decentralized distributed distribution. Both package managers deprive me of control over my system. I don't like the aggressiveness with which these projects are enforced. When Let's Encrypt switched to Spat as the only official distribution of their tool, I had to switch to alternative tool acme.sh instead of installing bloated snapd on my server.
Under some circumstances (not all), GNOME has a dependency on logind. If you're in one of those circumstances, you can use elogind as a drop-in replacement. That's far from having a dependency on systemd.
>Main purpose of both overengineered projects is to tie users to vendor's services (snapcraft, flathub, e.t.c.). Both package managers deprive me of control over my system.
I don't think snap lets you deploy your own repository, but for Flatpak this is very very wrong. Check here: https://docs.flatpak.org/en/latest/hosting-a-repository.html
For example, one of my Void flavored machines is using runit: 11 tasks; but a freshly booted headless Debian using systemd: 95 tasks... and that is with a massively slimmed down, purpose built, kernel. Abstractions atop abstraction... the amount of waste in the pursuit of generality has to stop - if this silicon shortage doesn't do it nothing will.
People didn't like the way systemd developers pushed the community to adopt systemd, specially when they asked for 3rd party developers to make systemd a hard dependency.
Unfortunately,people don't remember this today, and think users resisted to systemd adoption only because they didn't like systemd.
I find it much easier to reliably accomplish non-standard things with systemd unit definitions, than it used to be with the ill-defined complex set of shell scripts we had before.
systemd management of mounts can be similarly narrow about the types of configurations it supports, and systemd-firewalld just sort of openly only claims it can support simple use-cases.
systemd-journald is also a net loss of flexibility compared to rsyslog but, on the other hand, rsyslog could quickly turn into an inscrutable mess if you used any of the advanced features (rainerscript...), so this may not be an entirely bad thing.
And in general systemd is, well, opinionated. I hate to bang on resolved too much but I just happen to have spent quite a few hours last week figuring out resolved problems. Resolved does not handle "dotless" domains correctly in a lot of existing environments (it's very particular about exactly how the search domain is set up). Poettering has basically responded that it's because dotless domains are stupid and no one should use them, so it won't be changed. I don't necessarily disagree that dotless domains are not a good idea today but it does mean that resolved breaks a lot of older corporate and institutional environments that have been using them successfully for decades. This manifests as "I updated my distro and the intranet stopped working." That kind of breaking change is not very common with core Linux services and isn't going to make many friends in the IT crowd.
But, I am 100% agreement on the whole resolvd thing. It is a complete fiasco for anything but someone's idea of a standard network.
Even the most basic things, like waiting for DNS to be up before mounting a network filesystem has to be done by writing your own units if you don't want to just try and mount and hope for the best.
I’m sure there are many other examples.
Honestly, even after 10 years, I still have’t had a single positive interaction with it vs. traditional init. From what I’ve seen, old-school init handled (and continues to handle) the problems systemd “solved” more robustly and elegantly. I don’t get it.
Consider a situation, where your sh-backed SysV init was missing some function you required. Something between start-stop-daemon() or ifup(). You could throw a bunch of shell commands (note: most shell "programmers" are not capable of doing it properly; they do some '==' comparisons in /bin/sh shebang, don't handle errors, ignore fact that variables might contain "funny" characters etc. - but in general most of the scripts I've seen during my 24 years in Linux was simply badly written, meaning at least using bashizms). And then ...you watch out for every single package update that would overwrite your changes. Or you could polish them and try to push upstream.
Now the situation seems to be worse - you can't just edit some random file to add missing function, you need to write it in source C files and recompile. Not a way for quick and dirty solution.
But instead, you can simply write the same (s)hell commands in a script and call it from appropriate unit. No messing with thousand-lines scripts, no risk of overwrites, no need for upstreaming.
I’m old enough to remember the culture wars and even older to remember a well thought out approach to overall configuration in Debian given the available tools of the time.
Note I said “good” documentation - many Linux systems have lousy documentation and they lack an obvious, sanctioned way to do it. Again, that’s the problem.
Defining a straightforward configuration structure for that complexity is going to be insane. NetworkManager + Debian's /etc/network/interfaces (IMHO) seems to be a good fit: auto config w/ a GUI in NM in the simple case, and if the interface is listed in /etc/network/interfaces, you can do it anyway you want with Debian's system, with pre-up, up, post-up, post-down, etc scripts to your heart's content to setup whatever complexity you need.
NetworkManager is god awful in this regard. Even for simple desktop uses, if I want something more advanced than simply connecting to the Wi-Fi (even this requires finagling), then I'm fucked.
For example, I've been trying to find what it is that keeps overwriting /etc/resolv.conf.
After reading through various man pages and online forums, I still don't know. Instead, I just stick to one access point, and manually write to /etc/resolv.conf everytime I boot.
Compare this to OpenBSD: nothing fucking touches my /etc/resolv.conf. That file is mine. I write to it and other daemons read from it.
chattr +i /etc/resolv.conf
Wonder if it solved it. If not, what's the fuss.
What specifically is a pain? Have you used it recently?
Maybe my end-user desktop network requirements aren't complex enough but it's been pretty damn smooth in my experience for the last few years. I used to avoid network-manager with wicd, but now it seems pretty stable. I use the network-manager and network-manager-gnome packages to get the toolbar thingy in i3wm, it's very simple to use. I also combine it with wiregaurd using the wg commands.
Are you doing something more sophisticated like routing traffic? or something else in a server context? I literally can't recall having a single networking issue that was Debian's fault over the last few years, throughout using lots of different APs.
[1] I'm a developer who can admin a Unix system if forced to, but how to administrate Unix has changed at least three times since I first learned in the mid 90s, and I'm tired of the constant changes (shakes my old-man fists at the sky).
It's like somebody decided that the problem with Linux was it wasn't complicated enough. And then everbody just went with it.
We are talking 10 years ago, not 1975 here. Unless everybody is 25 years old on HN...
Do you have any links for these statements? My memory is not great but I think I would remember anything similar to this.
[citation needed]
I remember spending days reading the arguments on the Debian mailing list. There was no push from systemd devs. (Endless threads about upstart, how tweaking upstart would solve everything and it's already in Ubuntu, and a lots of questions about guarantees. Are openrc/upstart/systemd maintainers willing to guarantee this or that? And as far as I remember, there were no guarantees, obviously.)
Before that when the hwdb and udev merged into systemd, people got the pitchforks out, it was the decision by Kay, but obviously "systemd devs" were somehow behind it.
When Greg KH supported kdbus, somehow it was again those meddling systemd devs.
Somehow OSS veterans just lost their agency overnight and became puppets? Nah.
Random system breakages for stupid reasons.
And everything hardcoded the ugliest way possible so that it is impossible to have alternate systems like some embedded ones.
quote:
"What most systemd critics consider "bloat", I consider necessary complexity to solve a complex problem generically."
endquote
This is par for the course for what's essentially prototype-quality software that's trying to address a number of non-trivial issues - but let's not pretend that real improvement is not possible here, or that systemd-as-it-currently-is does not stray from what's often called 'the UNIX philosophy' or 'the art of UNIX programming', of which separating mechanism from policy is a key component.
Can you please elaborate on this? It sounds like a very desirable approach, but I don't really see it "in unix". (I can think of a few program suites that use processes and piping between them as the "protocol", eg. cups, the executable files in a directory that are then launched in a loop pattern is more common, but that seems to lack the policy part. Whereas OpenSSH for example lacks this.)
I don't want to hear anything about stable or production systems or debugging from arch maintainers. Every so often I forget the journalctl and coredumpctl command line flags and have to google how to do basic tasks.
[1] https://bugs.archlinux.org/task/73002
EDIT: and before someone asks me why i'm using it, it's because i lke tinkering and for the most part i can deal with these "outages" on my laptop.
I think I'm just used to the installer. I like doing everything by CLI. Ubuntu is always trying to push something odd like Snap, and I think Xubuntu is defunct. The Ubuntu installers are very opinionated on what kind of desktop they want you to have.
One of these days I'll just buy all the ThinkPenguin stuff and go to FOSS-only Debian with xfce. Debian has a great installer, too. Then I'll turn my desktops into cattle-style systems where I blow up the rootfs once a year for full system updates.
My experience with Debian system updates have been pretty good; I don't think I have ever blown up my system through them. (when running stable)
Unless you explicitly disabled it, you should have old packages archived in `/var/cache/pacman/pkg`. So you can simply downgrade via `pacman -U /var/cache/pacman/pkg/python-x.y.z`; no need to go to an (possibly untrusted) archive.
I have been using s6 on Artix and find it a great init system and service manager with a clear cut design goal of knowing where to stop. It does what it is supposed to do and no more - by design. It doesn't have the user friendly interface of systemctl but that is being currently worked on. If one is looking for an alternative to systemd, I think it is an excellent option to consider.
I'm currently exploring how to generate minimal service environments from Guix package manifests. Being able to spawn svscan as a child under the root supervision tree means I can create each environment with its own service directory, and never worry about cross-contamination from whatever else is on the host.
I _think_ I did something similar to what you mention, though I'm not familiar with guix. Analogous to systems, what I _did_ like about systemd was the user level service management. So I essentially kick off a user privileged supervision tree for my session with s6 with its own scandir and service db when launching X. It is not under the root supervision tree but lives side by side at this moment since 1) I wasn't aware of s6-usertree-maker at the time and 2) i wanted any user to be able to kick off user privileged services without root intervention (e.g. setting up usertree-maker)
I use phusion/baseimage-docker[1] right now, love every bit of it. Having the database, web server(nginx) and the java backend all in one place works beautifully.
The only pain point is having to write `runit` scripts for startup and shutdown for each service, hence the wish for Systemd inside containers.
One process per container is indeed strange in a lot of use cases, you lose the self-contained advantage of a container...
Having said that, personally (old school) I lost 2 or 3 hours yesterday due to not realising systemd was changing things I changed for a reason.
You might be forgetting how bad it was at first. In fact I'd argue it was worse than initially feared, and the developer's supreme arrogance did little to help.
This is one of my go-to reasons when people ask me why systemd is bad. Once it overrode power settings I had configured in xfce-power-manager. Mainly I wanted the lid close to do nothing on AC power and to sleep on battery. It started sleeping for everything after an update once. I think in the end I couldn't figure out how to get the exact desired behavior with systemd and I disabled sleep entirely.
The other reason I often mention is the ridiculous shutdown timer thing that makes you wait for minutes sometimes for some service to end.
The main advantage to systemd is that a lot of unit files were already written and packaged with stuff, including --user variants, and also that it has a user mode at all (OpenRC doesn't yet). Fairly minor advantage, imo. In general I prefer shepherd, runit, or openrc. My needs are pretty simple, though. Most of the time I just need to start/stop/restart a service, which is pretty simple in every init system I've used.
Not sure if serious? At that point, how is that any better than just using Windows? "Press Ctrl+Alt+Delete 7 times" has long been SOP there.
Color me as part of the anti-systemd crowd, but i still got to support it on machines where i didn't get to choose.
I didn't even know Windows did this. This is going to save my butt some day.
Is it something like a forgiving parser going rogue in an edge case?
The only thing I could think of is a systemd default setting that was added to logind.conf or something similar...that's why I'm asking.
My favorite quote (and when taken in context is much less tongue-in-cheek thank you might think):
systemd is an event-driven object manager with dependency-like side effects which ‘boxes’ primitive kernel resources and userspace subsystems into a generic object type called Unit. These Unit objects are scheduled through the state propagation mechanism of ‘jobs’ and dynamically dispatched via a singleton object called Manager, responsible for launching jobs in ‘transactions’ which do merging, cyclic ordering and consistency checks and serve as the main point at which unit dependencies are pulled in. Unit startup is executed as a non-indempotent parallel dataflow with weak ordering guarantees on the job level, mostly independent of the active state of dependent units.
Any night you have insomnia, just read that a few times and you'll be out like a light! :-)
“Idempotent” is one of the key words used as a cudgel in this way. It’s great for the bullshit artist, as to a layman the dictionary definition is completely inscrutable.
Idempotence is handy because it implies that one can retry safely if one isn't sure whether something succeeded or failed. Not sure if your file was saved? Save again! If it was indeed saved, the file continues to be saved. If the file was not saved, it is now. Yay.
I would kill for such descriptions of every system I work with.
surely it is the very opposite of bullshit.
This isn't written for laymen, who would obviously never be reading about what SystemD is. It's written for specialists who do engineering work on or with Linux, and who actually need to know the technical details of their tools.
And anyway, I suspect plenty of laymen, and highly trained professionals in other fields like law, medicine, and philosophy among others, would have no trouble understanding what "idempotent" is. It's not inscrutable at all, but rather is one of the easiest concepts in CS and software engineering to understand.
> “Idempotent” is one of the key words used as a cudgel in this way. It’s great for the bullshit artist, as to a layman the dictionary definition is completely inscrutable.
Are you saying that being "idempotent" is completely inscrutable? Is it not a fairly straight-forward concept?
What does it mean for a dataflow to be idempotent in the first place? What purpose does it then serve to describe a dataflow as non-idempotent? What they meant to say, I guess, is that the dataflow may contain multiple actions for the same unit, and that systemd makes no guarantees about the end state of the unit in such a case. But even then, it is the execution which should be described as non-idempotent, and not the dataflow.
And the description of systemd just uses bog standard terms related to what systemd does and the resources it manages.
But I get the feeling that both this and the parent comment it agrees with were more about the putdown and talking bad at someone, than about a technical or other objection.
And because it is such a common property when talking about algorithms and APIs the word is pretty common among CS people.
One very popular example are REST APIs because some HTTP verbs are idempotent and others are not [1]. This is also the reason why your browser asks you, if you want to submit a form again when refreshing a page that has been reached by submitting a POST-form.
[1] https://en.wikipedia.org/wiki/Representational_state_transfe...
In this case however, the words are all fairly precise and accurate (at least from my basic understanding of systemd). They might not be words used in everyday conversation, but they don’t need to be unless the document is targeted as introductory material (which I don’t think is the case).
That’s not to say this is the only way to explain it, but it is a very concise way, which I at appreciate.
I'd say the other mistake the article commits is by trying to paint this as being a very complex design for the problem space. It's not when you consider the design constraints. You want services to have dependencies, and you want the updates to the unit files to be committed atomically to avoid stale data. Hence, the dependency graph and the transactions.
Sometimes, a concise, high-level overview is what you need.
It really doesn’t.
Maybe because I've worked in embedded Linux for so long (including on inits of various forms), but that description clearly articulates exactly what systemd is and does and I appreciate the direct, crystal clear technical language rather than some long form essay written assuming a clueless reader with 0 technical background.
I expect everyone here commenting is familiar with systemd. I doubt very much that anyone who was unfamiliar with systemd and read that description would then have any idea what it does or how it works.
> I expect everyone here commenting is familiar with systemd. I doubt very much that anyone who was unfamiliar with systemd and read that description would then have any idea what it does or how it works.
It depends on what knowledge you expect. If someone is familiar with general init management and some programming concepts, this short summary will get them up to speed (and that's what I/we were arguing). If you grab a layperson of the street and read them this, it's obviously not going to tell them anything - but that's not the target audience.
What I value is a short summary which brings a reader, who knows the concepts and the space a software is operating in, up to speed quickly. And that's what this summary does.
- event-driven
- object
- side effects
- kernel resources
- userspace
- subsystems
- unit
- scheduled
- state propagation
- dispatched
- singleton
- merging
- cyclic ordering
- consistency checks
- dependencies
- non-idempotent
- weak ordering guarantees
And while this list is good for BS Bingo, as these terms are being used in other contexts too, the overall description of what systemd does is pretty precise for just three sentences.
Make doesn't do dependencies between unrelated projects. A makefile can't say "I want to run if X runs, but I should run before them".
Make doesn't have supervision, you'd need some kind of declaratively configured poller.
Systemd is also parallel, there's probably some complexity there that make might or might not handle well.
I think the most important problems would only be obvious once you started using a replacement system though. Systemd almost totally replaces the need for any imperative code whatsoever for process autostart, configuration, and supervision, and that might be hard to do in a more modular setup.
You don't need extra stuff besides the service file. Everything is extremely consistent and done the same way for every service.
Polling is better-addressed by having each service implement its own error handling. Restart loops are the wrong policy for most services. In the rare cases where they are actually the correct behavior, they’re trivial for the service author to implement.
Those that don’t understand make are doomed to reimplement it poorly.
They're not exactly trivial. There's restart delay, number of times to restart, passing stdout through so the journal can get it, and there's failure reporting.
How is PID 1 supposed to know if my service failed when the restart daemon is still up and running, but my actual app keeps crashing? How can it tell the difference between my app and a supervisor, or a fully failed app or just one failed subprocess that my app can handle? Now you need some scheme for that.
If PID1 doesn't know what's failed, how would I get a list of all failed systems programmatically?
Systemd could provide a standard supervisor component, but then it couldn't ever do anything but restart failed apps, or you're back to people complaining about monoliths.
You'd have to have your service file explicitly call it with some command line args making your syntax uglier.
And more importantly, it might be harder for the systemd dev team because now they have a communication protocol between this restart daemon and everything else.
Or they could leave restart daemons up to the user... and everyone would probably choose 3 different ones, that might or might not keep up with compatibility. That would break the "One and only one obvious way" principle.
When you start splitting stuff up in tiny bits, you get more overhead, all the extra nice features seem to evaporate.
INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
I may try it again one of these days now that the latest Devuan stable is based on Debian Bullseye.
The "why" is simple: to restore some balance in the force.
I do prefer my Linux world to be one where I still have a choice as to which init system and PID 1 my system is running.
https://lwn.net/Articles/786593/
It's not even the immaturity of the joke that bothered me back then, but that the response to the criticism was to imply that it is "corporate bullshit" and/or "giving up on having fun".
Dependencies in make are equivalent to systemd's Requires= and After= combined. Systemd is far more general in this regard, both in that Requires= and After= are separate concepts, but also in that there is Wants=, BindsTo=, etc. and their inverses (RequiredBy=, Before=, etc.). Sometimes when I'm writing Makefiles I wish Make were capable of expressing some of these concepts.
I think the core of the problem was somewhere between the style of introducing changes and the way how people communicated. After all, it speaks for itself that pulseaudio and systemd shared a lot of common criticism. Both are technically sound systems today, but in a way, systemd was just the escalation of the pulseaudio debacle.
A very small percentage of sysadmins really care about what init system is running on their servers. If you look at who's forking distros and making systemd-free variants, and look at who the vocal anti-systemd users are, they're overwhelmingly desktop-focused users. But this is strange because desktop Linux barely needs an init system at all. The long-running daemons on a desktop Linux system only need be started on startup, if they crash, restarting them is unlikely to leave the system in a workable state anyway.
However, because server Linux dominates the Linux ecosystem discourse and distro news, people have an inflated view of how important these components are.
So my take-away has been: it's safe to ignore systemd criticism, since it's overwhelmingly coming from hobbyists who have no real use for an init system anyway.
I have had enormous issues with systemd, not when using it for desktop purposes, but precisely when using it for server applications.
As an example, systemd can't handle a service which depends on a specific network interface (or specific bind address) before being brought up and the systemd developers blame the software for not listening to rtnetlink events or not relying on linux specific socket options[0]. This is unacceptable, it makes it incredibly difficult to run systemd on hosts connected to multiple segregated networks without lots of dirty tricks involving firewalls.
[0]: https://www.freedesktop.org/wiki/Software/systemd/NetworkTar...
Also, don’t get bogged down by systemd-* services, they are related but are usually not part of systemd at all.
I'm generally a fan of simple systems that don't do much controlled by simple text files. Maybe systemd is way better at runit for more complex workloads than a desktop or a simple server. But maybe it isn't. The intersection of people who like systemd and have tried runit is probably quite a small one!
See lively discussion from last time: https://news.ycombinator.com/item?id=23062072
(but also, it's true, people tend not to use HN search before posting)
But you have probably thought of this already and didn't do it for a good reason.
In this case I hadn't seen it before so it was news to me.
But I guess making it too wide would create false positives.
Systemd, ten years later: a historical and technical retrospective - https://news.ycombinator.com/item?id=23062072 - May 2020 (432 comments)
and:
Systemd, 10 years later: a historical and technical retrospective - https://news.ycombinator.com/item?id=23204666 - May 2020 (59 comments)
Back when I had linux as a desktop (around the day of slackware 10) there were plenty of problems with simple stuff like mounting usbs, connecting to wifi etc--possible but a pain. I've hardly looked at systemd or linux in such a way for a long time, however.
My preferred WM is evilwm. I read the entire source to it in a few hours, and occasionally edit the source when I want to customize it for my odd workflows. I don’t think that’s possible with wayland.
Also, wayland forced a bunch of questionable security policies around screen capture and keyboard handling, and didn’t provide escape hatches required for basic functionality. Is that fixed yet?
I stopped paying attention years ago, to be honest.
KDE and Gnome have their own wayland implementation but this isn't really a problem. You can still write you minimal WM because their is a project called WLRoots which does the same job as the old x11 package. It lets you trivially interact and be compatible with the wayland spec without having to implement it yourself.
Screen capture and keyboard privacy is essential for the future of linux IMO. I don't know the exact state of it but I did not have any issues screen sharing with wayland compatible apps. There were some issues with x11 only apps but I think PipeWire solved that.
Well, the wayland protocols are very poorly documented compared to X. But most WM's have to support a bunch of obscure X extensions to get a reasonable featureset, that bring in much the same complexity as Wayland.
The security policies are not questionable. XServer is beyond fixable. A single rogue extension can screen share everything you do to a remote server, you would not even recognize that.
I am actually optimistic that in another dozen years or so Wayland might be pretty nice, but I certainly wouldn't consider it an examplar today.
SystemD is essentially forced (for better or worse, not gonna argue) because it is a hard dependency for certain software and tightly integrated with other tools in the systemd ecosystem
It is opaque because I cannot read, modify or understand what it’s doing without breaking out gdb, heaven forbid I want to alter its behavior in a less than approved way.
This leads to weird situations like the MySQL unit executing a bash script to run: because systemd has limited preflight checking compared to alternatives.
Before someone claims I want sysvinit back: No, I don’t want that. I would simply prefer systemd to be smaller so that we can replace it with something better or more tailored should the need arise.
We’re locking ourselves in to prototype level software (which we best into working semi decently) with obstinate leadership team who deny any faults.
In this respect I think that systemd is really a victim of the Unix kernel/shell style, where 'kernel' is read expansively to mean 'everything in C' and 'shell' is read to mean '/bin/sh'. The problem is that where shell is easily examinable and understandable for most technical folks, compiled C simply isn't (yeah, yeah, we all know someone who can debug in assembler; that guy ain't me).
It would have been nice to have had one system, with a typesafe compiled language. Scrutable, fast and safe. We have an example with emacs: one can always just M-. to a function's definition (even in the C core …).
It's a real pity that systemd hadn't been written in Common Lisp or some similar language.
Perhaps at some point they'll switch to something that's established instead of maintaining their own IPC protocol.
- GDBus, if you're developing for GNOME
- QtDBus, if you're developing for KDE
- sd_bus, if you're writing low-level C
- zbus, if you're using Rust
For other languages, you'll want to look for bindings to one of those libraries, instead of bindings to libdbus.
Excluding the kooks angered enough to send hate mail to the systemd developers, I can understand the old criticisms and aggravation from sysv folks. I was front and center as a relatively new user of Fedora at the time systemd was adopted. While I was familiar with sysv, I wasn't attached to it. I also found sysv a lot more confounding and obscure than systemd in the early days, even though the transition did have rough spots. But I'm definitely attached to systemd now.
[1]Linux Service Management Made Easy with systemd:
https://www.packtpub.com/product/linux-service-management-ma...
- The developers had a condescending approach and ignored the wider community.
- Simple things like log files no longer being plain text. Obtuse.
- Arguably designed to inject unneeded complexity that will cause money to flow into big orgs. 1.2 million lines of code vs shell scripts. Just wow.
Wow. I had forgotten how ropey early Linux was. I mean, I rip on Windows folk about early versions of Windows prior to 98SE or NT4 (and can recall having to install Winsock on a home Win95 install and thinking why the hell????) but finding out 30 years later that my early struggles with Linux were not entirely down to my own incompetence is ... refreshing. I mean how do you Google (ok in those days Altavista!) a "how to" if the network stack is borked? How do you even download the drivers or updates if the interface fails to come up?
It sounds silly but I'm so glad to know it wasn't something I was doing wrong on my end. 30 years later and dayamn!
k3s could stop everything, but it doesn’t. It’s a design decision.
Chances are the k3s service unit specifies something else (like 'none') - whether or not that is actually wise would probably require some research.
At one point some distributions accidentally configured the sshd or cron unit to do that. People were a bit surprised when restarting sshd or cron killed every single background child process.
See https://www.freedesktop.org/software/systemd/man/systemd.kil...
Not to mention if k3s is just managing containers running under docker or containerd, those processes are not parentless just because k3s stopped.
Under crio to stop all containers you can (well, should but YMMV depending on any changes we’ve made recently)
systemctl stop kubepods.slice
Note that doing that a subsequent start shouldn’t bring the containers back unless k3s and crio were both running (the transient units don’t survive restart).
Note: there’s tons of complexity here that has changed over the last five or so years as container runtimes evolve, so if anything I said is wrong now it was right at one point and might actually be a bug.
Kube for instance didn’t use to gracefully shut down containers on node reboot, and over last few years we’ve gradually fixed all of the complexities so that you do get the correct shutdown behavior between systemd (which takes the “power down” input) to the unit (which encodes how long graceful shutdown should take based on the pod termination grace period seconds) to cri-o (which has to set the correct behavior in systemd) to kubelet (which has to stay running to keep updating the API) to the other components on the node (like container networking, which needs to stay up until all pod network pods are done).
systemd makes a lot of that a lot less painful than it would without.
Now k3s of course starts tons of stuff, mostly in containers. Therein lies the problem: By default, Systemd indeed tries to get rid of the entire process tree. It does this using control groups (cgroups). Unfortunately, it is easy for a process to “escape” its control group, at least for root.
Stopping the master process could absolutely be made to also stop everything.
It would be great if k3s did properly integrate with Systemd. It doesn’t, but Systemd is not to blame for that. That’s my point.
If systemd is your init, then it manages everything. It may not manage them at a service level ala systemctl stop, but that's true of pretty much everything. Whether or not k3s brings down everything with it is on k3s. I certainly see why stopping k3s might not stop all dependent processes: it's useful for live upgrades.
However I quickly learned and then it was mostly a non issue.
I was confused when years later there was still push against adoption by other distors.
Lots of people are not surprised when you tell them that pulseaudio was created by the same author. Because it is kind of working in most cases. But there are always, aside, a lot of annoyances or things that work badly without a real justification: hard to fix and understand what is going on when it doesn't work; sometimes it decides alone to not use or not show the output/input that you want and you have no possibly to decide; sometimes it is using 10% of your CPU despite the fact that you are not playing a single sound at the moment,...
I don't care how it works and didn't take part in condemning it back then either, and I think that probably it made something better (? not that I felt it), but from a pure sudo-user perspective it is a strange ini-based tech-ridden(?) launcher to me. I've managed small networks before it and haven't really experienced any trouble that it supposed to remove(?).
Tldr: I still don't know what it is and what is it for. Servers worked before and after it, maybe that is already a good experience.
journalctl -u custom-daemon.service
Also, you can store plain-text logs as well if you prefer by exporting journald’s output.