Systemd: The Biggest Fallacies | Hacker News Reader

Systemd: The Biggest Fallacies

judecnelson.blogspot.com

95 pointszdw50 comments

First of all, thanks for linking to uselessd.

The writeup was quite nice. I was actually in the process of writing my own notes to respond to Poettering's "The Biggest Myths", but your approach is better. I'll definitely use it as a reference to link to in discussions.

That said, I have a little caveat for #9. Though systemd violating KISS is virtually undeniable, you should reword it so as to point it out on systemd's own merits, not in relation to sysvinit, which systemd explicitly intends to be more complex than.

I have absolutely no idea why this comment has been voted to the bottom of the page. Friendly, innocuous, and thoughtful.

The systemd promotion brigade has invaded hacker news.

> First of all, thanks for linking to uselessd.

No problem :) I'm looking forward to seeing how it turns out!

> That said, I have a little caveat for #9.

My reason for including this fallacy was that I've seen it argued by well-meaning people on multiple occasions. I take you point, though, and will clarify this.

He's missing the point about socket activation. The problem it solves is different: if you have daemon B that depends on daemon A, you want to express that dependency in the init system, so that it starts A before B. But just starting process A isn't enough to ensure that A is actually listening on the socket that B wants to connect to, since it takes some time to load the binary and do initialization. There's a race condition there, where B can try to connect before A binds to the socket. To fix the race, the init system needs to either monitor for the availability of the socket, or do it systemd-style where it opens the socket itself.

As a bit of a disinterested observer (I'm mostly in the "just make it work" camp), I can't help but feel that if you pile up the problems that systemd seems to solve on one side, and pile up the enormous piles of code that are apparently necessary to solve the problem on the other, that systemd does not come out looking too good.

(Writing this in C was probably a silly idea.)

what language do you think systemd should have been written in?

Honest question, how frequent or serious of a problem is this in practice? Most of the servers I deal with have only a handful of daemons running concurrently, and of those, maybe only a couple of daemons have interdependencies, and of those, they don't seem to have much trouble polling each other and coordinating startup. I mean, from the software developer's perspective, it's probably not fun to write that polling code and get it exactly right. But from the sysadmin's perspective, this doesn't seem to be enough of a real-world problem to justify the adoption of systemd.

(I am not a fan of systemd, but I'm being neutral here because I won't learn anything if I dismiss it -- so hopefully someone will give an equally fair response.)

To be honest, I don't think I've ever tracked a problem down to that race condition. But I'm willing to believe some of the things that systemd does (and increased parallelization in general) might increase the likelihood of it occurring.

I think the larger point to take away here is that the systemd authors have a bias for doing "the right thing"[1], even if it means additional (implementation) complexity. Unix has traditionally been a stronghold of "worse is better" culture, which I think is where all this friction is coming from.

[1] https://en.wikipedia.org/wiki/Worse_is_better

That's for desktops where boot times are more important. Starting A and B at the same time allows to lower that time.

On servers you want to wait for A to be ready before starting B. Otherwise the socket min timeout from B to A is dictated by the startup time of A and not the normal runtime timeout (seconds vs millis).

In my recent experience, server boot time is more important to me than desktop boot time. My desktop and laptop basically never boot anymore, they just suspend and resume. But I wait for ec2 instances to boot at least several times a week.

Giving me further proof that the "shism" is between traditional server admins and cloud admins.

Except that I am a cloud admin

Most of the EC2 boot time is taken by the resource allocation and provisioning of the host (~2min). Changing the init system would only save you a couple of seconds of that.

Some of these are arguments I'm glad to see getting more attention, such as Fallacy #1: "Systemd is multiple binaries, therefore it is not monolithic".

Others strike me as a stretch. For example, Fallacy #4.1: "Unit files reduce complexity". No, I don't want the least complex init system init system possible. I think it's obvious to people who have written both system v init scripts and systemd or upstart configurations that the latter are dramatic improvements. The chance that I will write a buggy init script is, unfortunately, high. The chance that I will need to debug systemd when writing a unit file is very low.

Really the only "monolithic" problems I see are with journald and logind. For the other pieces people don't want to use them and they don't have to. The fact that e.g. networkd can't run without systemd is fairly irrelevant since no one is asking for that; they're asking to not run networkd which works fine.

This. I would have been happy with anything that turned initscripts, which are often hundreds of lines, including tons of decisions and variables, into a simple declarative conifguration file. systemd seems to do OK at that, though I am vaguely uncomfortable with how tightly coupled things are in a systemd system.

>I think it's obvious to people who have written both system v init scripts and systemd or upstart configurations that the latter are dramatic improvements

Except that is a false dilemma, which he points out. Sysv init takes more work because nobody has bothered to fix it. You don't need a massive complex set of software like systemd to make this easy, see the BSDs init scripts like he suggests.

That doesn't really invalidate the argument. Upstart and systemd units are dramatically easier and less bug prone than the existing sysv alternative. If it was nontrivial to fix they would have.

The missing beef here is actual bugs and problems with systemd. Post some zero days... Nobody challenges that businesses with paying customers all indepdently came up with the need for smf, launchd, upstart, and systemd. Are they just adding shit to add it or do the see value and need?

> If it was nontrivial to fix they would have.

There is no basis for that assertion. It is trivial to fix, again, see BSDs. Just because nobody bothers (or people bother and their effort is rejected due to politics) doesn't mean it is nontrivial.

Apologies if you already knew this, but you may be interested in pleaserun[1] which attempts to manage the differences in init formats from a common interface.

It's still quite new but I've been reasonably happy with it so far. I did have to extend it a little to allow commands like bootstrapping for clustered software.

[1] https://github.com/jordansissel/pleaserun

Pleaserun is a much-needed project, I would recommend that people use it as a starting point for generating init configurations (although unsurprisingly the sysvinit script it generates has portability issues [0]), and there are plenty of situations where what it generates will be good enough.

I'm warming up to the idea that configuration management systems like chef should use pleaserun or add a similar abstraction themselves.

[0] https://github.com/jordansissel/pleaserun/issues/49

There is a lot to digest, but so far this this looks like an excellent example of critical thinking, regardless of one's feelings or intuition regarding systemd. Both proponents and detractors have much to be thankful for in this article.

Some of his arguments look a little bit botched.

For example: > Fallacy #1: "Systemd is multiple binaries, therefore it is not monolithic"

Well following Wikipedia (http://en.wikipedia.org/wiki/Monolithic_application) means either a non-modular application, or a self-contained application. In this sense either his counter-argument is simply wrong, or the examples list he gives at the and is wrong.

> Fallacy #4.1: "Unit files reduce complexity"

Here he compares the ca. 275.000 LoC of systemd with the 10.000 LoC of shell scripts used as initscripts on Debian. Lets ignore that the 275.000 LoC contains much more than the unit management in the systemd daemon. But why are the bugs always in the scripts and never in the shell as he claims? And sorry, I don't take his word that C is always more error prone than the shell. This is true when the shell is used for its intended purpose: to start commands, pipe btw them and having a little bit of flow control. For everything else you surely want to have a general purpose language.

> Fallacy #7: "Systemd gives you socket activation!"

Well, according to boot charts, socket activation is not only a marketing gimmick.

EDIT: Add Fallacy 7

I think we need a separate term for "internal modularity" (Linux, Systemd) and "external modularity" (Unix, the Internet).

The former depends on internal interfaces, where individual modules help separating concerns but can't be swapped out or reused in other contexts, whereas the latter relies on formal interfaces, allow components to be swapped, and promote reusability. (Also, the former can be monolithic or not, but the latter can never be monolithic.)

"Unixy" is sometimes used, but it's not a great term.

270k lines sure do a lot more, but I don't want to do more, I just want my machine to boot.

If it doesn't boot I want to debug shell scripts not muddle through thousands of lines of C dealing the time syncing or socket activation.

"shell scripts are not inherently buggy" not the week to argue that point.

It's not shell scripts, it's /bin/bash -- the guy fieri of shells.

Which also happens to be the default or expected shell in the majority of sysv installs on Linux. '/bin/sh' symlinks to bash, shell scripts specify '/bin/sh' So it's another example of a distinction without a difference to single out bash. Most sysadmins cannot just switch to ZSH to to write their init scripts.

Debian and its many derivatives (including Ubunutu) use Dash, and the BSDs tend to use some form of Ash. I don't have the statistics but, if it's true that a majority of distros still use Bash for noninteractive use, it's a tiny majority.

Well, a lot of them then... I know that Arch and Gentoo at least use Bash and I'd be surprised if they were the only ones.

Arch is using systemd and openrc comitted to using POSIX sh in their init scripts some time ago. So you can use dash or whatever posix compatible shell you want with it.

On Ubuntu, since 6.10, Dash has been the default system shell rather than Bash. '/bin/sh' symlinks to '/bin/dash'.

You've already been corrected with regard to the rest of your argument, but I'll correct you on the last part too: they could do so just fine. That's the power of a modular, non-tightly coupled system. Any script that uses ksh, zsh, bash, csh, tcl, lua, rc, whatever, runs fine under sysvinit.

No, my point is that you cannot just take a script written for bash and run it on Zsh or Ksh and expect it to just work. (Well, sometimes you can, but that's beside the point.) Yes you can write your init scripts in whatever language you want, even Lisp or Python, but you have to do that for every daemon you switch over. Imagine a lot of these Linux distros which often implement their own init scripts for each program they support having to switch shells. That's hundreds of scripts to rewrite, or at least proofread and test.

And nobody does that. They write for sh, that qll those others are compatible with.

This means you can swap one part and expect things to work, rather than having to swap the whole engine because you want to swap one part.

It's not just bash. Shell scripting in general is "stringly typed" and brittle because of that. Furthermore, when an error occurs and no special options have been set, the script will just continue as if nothing happened. This behavior does not encourage writing correct programs.

> Fallacy #2: "Lots of people use systemd, therefore you should too" > This is a classic example of the bandwagon fallacy.

On the other hand, network effects are real. You'll have an easier time finding help if the system you use is used by many others.

... and in this case systemd is popular not because users decided to use it, but because distributions decided to use it, which means that systemd is going to have a stronger support network and more developer attention than a system which is not popular.

Good support and active development are pretty big advantages.

"For Internet socket activation, all you have to do is start your daemon the old way, have it bind on the port, and let it sit there idly. If the kernel needs the RAM, it will swap the (idle) daemon to disk. If the kernel receives an incoming connection, it will swap the daemon back into RAM so it can handle it. Systemd doesn't need to be involved at all here."

So you want me to patch my Postgres instance to implement this? Or should I just do it in the init script? I guess that's possible, but it requires your sysadmin also be a programmer.

That's how daemons normally work. The author is obliquely pointing out that socket activation isn't that much of a benefit. Historically, "socket activation" using inetd was common but as hardware got cheaper people gradually dropped inetd in favor of just starting all their daemons at startup.

I understand the author's point; he's not being oblique at all. I just disagree. He explicitly claims that the kernel providing socket functionality is enough. But it's not. The example using mkfifo for example, is terrible for instance. It provides no lifecycle management and you'll end up launch multiple versions of a daemon if you accidentally send multiple signals to the pipe. There is a huge difference between "this is technically possible," and this is doable in a way that doesn't have many unforeseen pitfalls. Systemd asks much less of professionals than sysv and that's it's chief selling point.

The postgres daemon already works that way. Nearly every daemon that listen to a socket works that way (a few don't, surprizingly).

That's the point.

I am getting so sick of these blogs for or against systemd.

It is free software. If you do not want to use it, use a distro without it. And vice versa if you want it. And if you don't like that Debian or Canonical or Arch a few years back or OpenSuse have switched to using it... too bad, you are not the maintainer of the repos, and if you were an active member of the community with voting power, you were outvoted. That means a majority of people with stake in these distros wanted systemd, and it is their project.

Make your own distro without it if you don't want it. Or use one of the dozens of distros that don't use systemd, or one of the forks that have been made to remove systemd already.

It's obvious you didn't even read the article.

EDIT: I don't know the motivation for the downvotes. I replied like this because the commenter posted a canned cookie-cutter reply that is addressed in the article as Fallacy #6.1.

>I don't know the motivation for the downvotes

It's pro-systemd brigading. The cancer is spreading.

Terrible logic. With this kind of logic anything can be justified.

Looks like 'you can do it, or you can not do it, or somebody you depend on can do it, and if you don't like it you can leave, everybody has made the decision and you have lost, get out, yadda yadda' has been deemed a good argument in this thread. Scary.

Well, following your line of reasoning: If you don't like these articles, too bad! You have been outvoted. Go build your own news website from scratch.

Reply on news.ycombinator.com