Hacker News Reader

Comment by weland | Hacker News Reader

Unfortunately, the systemd development team is not that easy to work with, which is part of why a lot of people distrust systemd the way they do.

In theory, yes, it should be the way you mention it. In practice, even bug reports end up unacknowledged and labeled as features after a few philosophical debates.

No one who has used more than one Linux distro would disagree that a unified system to control init services is a good idea. It's this particular implementation that's problematic, not the principle.

View on HN

One lovely example i have seen is Poettering completely misunderstanding a perfectly valid use of su, and instead recommending that people ssh back to their own box if they wanted a shell under a different user.

This came up in relation to a systemd set environment variable that would break software by overwriting one user's config files with that of another's.

The variable was apparently set that way because of pulseaudio.

The really worrying part was any lack of "mea culpa" from Poettering once someone outside the bug report told him that pulseaudio could be changed so that the variable was no longer needed and Poettering implemented that change.

Instead he just left the bug report hanging, and others had to report on the changes and declare the issue sorted.

But I think you could find a vocal minority (or several) to oppose ANY particular implementation. One of the reason the anti-systemd crowd has such a hard time gaining traction is because the only thing that unites all of them is opposition to systemd. They don't agree on what should be in its place instead -- some want to hold onto sysvinit, some want Upstart (although that's a dead end now), some want OpenRC, some want dmd, some want runit, some want something else or something that doesn't even exist yet.

Here is the thing. The problem is not systemd as init, it is systemd as, well, systemd.

The constant addition of daemons and functionality to the overall systemd codebase by either subsuming or replacing existing, INDEPENDENT, projects.

As such, who cares what init is used. The thing is that until systemd the other daemons/services started by init didn't particularly care what init it was.

Previously you could piecemeal your changes. Didn't like syslog? Grab something else that talk the lingo and slot it into place. Sorry, but journald demands that the init is systemd.

Want something to handle seats/sessions? Sorry, but logind only works with systemd as init.

Etc etc etc!

Going on and on and on about inits is a smokescreen.

It's very easy to agree that nuclear pulse propulsion is a bad idea for a vehicle, even among people who otherwise have very different preferences on what kind of engine they like on their vehicle.

Unfortunately, yeah, the whole debate there is unproductive. Instead of focusing on real problems, it gets stuck into an endless debate on Unix philosophy, carried among a bunch of people who aren't really sure why it's a problem if something doesn't follow it and another bunch of people who insist that they're following because saying otherwise would be bad PR.

> It's very easy to agree that nuclear pulse propulsion is a bad idea for a vehicle

Speak for yourself. Who cares if my rocket is blanketing Earth in nuclear fallout if I get to ride it to Jupiter? :)

    In practice, even bug reports end up unacknowledged and labeled as
    features after a few philosophical debates.

Where is the evidence?

All over their bugtracker, but the most famous one is here: https://bugs.freedesktop.org/show_bug.cgi?id=76935

An excellent example of a failure to communicate.

Linus said that the fact that systemd parses the "debug" parameter is not a bug:

http://lwn.net/Articles/593677/

But sadly that didn't stop various Linux developers from bullying the systemd developers in that bug report instead of trying to find some common ground.

The actual bugs (assertion failure creating debug output in systemd, and inadequate buffering in Linux's /dev/kmsg) could have been fixed much faster without the pointless flame fest.

Your link supports the assertion that systemd doesn't play well with the community -- Linus goes on to say:

    It does become a problem when you have a system service developer [Kay] who
    thinks the universe revolves around him, and nobody else matters, and
    people sending him bug-reports are annoyances that should be ignored
    rather than acknowledged and fixed. At that point, it's a problem.

Which was ultimately caused by a bug in the kernel, and the systemd developer ultimately accepted the proposed changes just to avoid any kind of confusion (to me, it looked perfectly acceptable to parse the kernel command-line, multiple applications still do for other purposes).

Arguably, the tone of the discussion resembles more the usual LKML tone rather than the one observable on the systemd mailing list.

If this one isn't satisfying, digi_owl has a nice one, too: https://news.ycombinator.com/item?id=9723476

Never heard about it, is there any link with more details?

https://bugzilla.redhat.com/show_bug.cgi?id=753882

Enjoy.

Interesting read, thanks!

However I would not characterize that as "Poettering completely misunderstanding a perfectly valid use of su".

It seems a quite complex issue about a workflow that is only half-working even without systemd. A systemd patch has been committed that fixes the most blatant error, but of course it doesn't solve all problems since most of them existed before systemd.

I don't know in what corner of the universe was that "only half-working". It was working absolutely fine. It has been working fine since... I don't know, v7, when it was first introduced.

The few times I tried to run GUI programs through su I always got plenty of warning about DBus and stuff. But I don't do that very often, so that's all I know.

I fail to see how systemd hijacking a kernel boot argument for its own purposes is a "bug in the kernel".

It worked before, it stopped after a change in the kernel, another change made the problem disappear: the root of the problem was definitely a bug in the kernel.

That said, it's perfectly fine for the kernel developers to clarify that a flag is for their own exclusive use, and indeed it seems that this is how things ended up. Note that there are multiple userspace applications parsing the flags on the kernel command line: the most striking example when discussing the "debug" flag is the "quiet" flag.

> That said, it's perfectly fine for the kernel developers to clarify that a flag is for their own exclusive use

Oh for fuck's sake. It's a parameter passed to the kernel. Programs shouldn't parse it for precisely the same reason why Firefox doesn't parse sshd's arguments. I.e. because they're passed to sshd, not to Firefox.

It's a ridiculous clunky hack that stinks of large company coding practice: what they actually needed was a way to pass arguments to systemd itself.

The logical way to do that in an OS that has to be portable and flexible is to implement a mechanism for the kernel to pass some of the arguments it receives to init. Because there's no reason to assume that init is going to be systemd, that there's going to be a procfs ready for it so as to be able to read the kernel's parameters, or that the init process can read it for that matter.

The way the systemd development team chose to do that (widely seen in many other projects from the same lineage, like Pulseaudio) was ha look at that we can see the kernel's arguments so how about we use those and what do you mean "this can have unwanted side-effects", it works perfectly fine on my system.

The fact that the developers (again, it's a known pattern) decided to play the PR game and call it "expected behaviour" doesn't make it less of a bug. It's also expected that a division by zero will crash your program. It doesn't mean that blindly accepting any input values is not a bug, nor that the correct way to solve "your program crashes if I pass 0 as the first argument" is "well stop passing 0 as a first argument ya evil bastard."

> It's a parameter passed to the kernel.

Just like "quiet", which is totally meant to affect userspace programs.

> It's a ridiculous clunky hack that stinks of large company coding practice: what they actually needed was a way to pass arguments to systemd itself.

Having a global "debug" knob would have been useful when one does not know if the problme lies in the ramdisk, plymouth, systemd, udev, lvm, whatever. Having a systemd-specific flag defeats that purpose.

> The logical way to do that in an OS that has to be portable and flexible is to implement a mechanism for the kernel to pass some of the arguments it receives to init. Because there's no reason to assume that init is going to be systemd, that there's going to be a procfs ready for it so as to be able to read the kernel's parameters, or that the init process can read it for that matter.

The kernel command line, readable from procfs, has been used for this purpose (see the "quiet" flag) for a very long time now.

> what do you mean "this can have unwanted side-effects", it works perfectly fine on my system.

It worked perfectly on every system until the kernel bug triggered the feedback loop.

> The fact that the developers (again, it's a known pattern) decided to play the PR game and call it "expected behaviour" doesn't make it less of a bug.

It was "expected behaviour" in the presence of a kernel bug, which is why it was obviously not generally "expected behaviour" as you seem to imply.

> What userspace programs interpret the quiet parameter? > Genuine question. AFAIK, quiet does (or should do) nothing other than disable some of the kernel logging.

At least plymouth and probably every init system out there.

Linus said that "we very much expose /proc/cmdline for a reason. System services are supposed to parse it, because it gives a unified way for people to pass in various flags. [...] And yes, that does include "quiet" and "debug". Parsing them and doing something sane with them is not a bug, it's a feature."

http://lkml.iu.edu/hypermail/linux/kernel/1404.0/01488.html

> The bug report makes it immediately clear that this is not really about a technical problem that ought to be solved - no, the dirty user-space programmers should get their stinking paws off of our good kernel command line flags, how dare they!

You're right that Kay was wrong to close the bug immediately instead of trying to help finding the real cause, and you're wrong when you say that userspace should not parse the kernel command line, see Linus' remark above.

> There's no reason to even assume procfs will always be there

To the contrary, dropping it would be an ABI break so it's really safe to assume that it will be always be there.

> who's been programming for more than an year

Please, don't lecture others about their job. I'm not assuming you're an incompetent fool, please don't assume I am.

> if a program did that on Windows

And can we try to stay relatively on topic? Windows is definitely unrelated.

(I'm not otherwise replying in detail because you assumed that parsing the kernel command line was not ok, while Linus officially blessed its usage. Also we reached the max thread depth on HN, which should probably suggest something.)

> Just like "quiet", which is totally meant to affect userspace programs.

What userspace programs interpret the quiet parameter?

Genuine question. AFAIK, quiet does (or should do) nothing other than disable some of the kernel logging.

> Having a global "debug" knob would have been useful when one does not know if the problme lies in the ramdisk, plymouth, systemd, udev, lvm, whatever. Having a systemd-specific flag defeats that purpose.

Yes, and it would have been very useful if the systemd team had offered to implement such a feature, instead of silently pretending to have it.

That thing was news to everyone. Except Kay Sievers, apparently, who chose to share the big news with the world when closing the bug report.

> The kernel command line, readable from procfs, has been used for this purpose (see the "quiet" flag) for a very long time now.

By what non-broken programs? There's no reason to even assume procfs will always be there, let alone that the kernel will expose its boot parameters through it. AFAIK, quiet doesn't affect anything but the kernel's logging.

With obvious exceptions (e.g kernel debugging tools and the like), that shouldn't happen.

> It worked perfectly on every system until the kernel bug triggered the feedback loop.

No no. It worked perfectly because the systemd team never bothered to see what happens when the kernel actually tries to use the debug flag, too (i.e. when an assertion actually fires). It should be obvious to anyone who's been programming for more than an year or so that, when you start parsing arguments that aren't yours (which is bad enough in the first place!) you should at least ensure that you don't break the program that uses it.

Yes, the rate limit for /dev/kmsg was increased when it turned out there was a potential to incapacitate the system by flooding the kernel with data from userspace. For what it's worth, if a program did that on Windows, every half-assed heuristic detection engine would have promptly flagged it as malware.

> It was "expected behaviour" in the presence of a kernel bug, which is why it was obviously not generally "expected behaviour" as you seem to imply.

How the hell is "I'm passing an argument to the kernel and a program from userspace decides to interpret it" expected behaviour? If I pass rw, too, should I now expect mount to mount every system as read-write?

http://lkml.iu.edu/hypermail/linux/kernel/1404.0/01537.html

That says nothing about why the systemd folks are entitled to hijacking a kernel flag instead of (more sanely) namespacing their own debug flag, which is the actual bug that was filed (and in turn met with "it's not a bug it's a feature, so go whine to the kernel folks about the thing of theirs that we broke and don't feel like fixing", as if the systemd folks have become one with the Microsoft).

Whether or not there are other underlying issues stemming from such misbehavior is irrelevant to the fact that the misbehavior exists. It's like sticking your hand in a fire, then blaming your hand instead of your own stupidity for your third-degree burns.

> That says nothing about why the systemd folks are entitled to hijacking a kernel flag instead of (more sanely) namespacing their own debug flag

"No, we very much expose /proc/cmdline for a reason. System services are supposed to parse it, because it gives a unified way for people to pass in various flags. [...] And yes, that does include "quiet" and "debug". Parsing them and doing something sane with them is not a bug, it's a feature."

http://lkml.iu.edu/hypermail/linux/kernel/1404.0/01488.html

> which is the actual bug that was filed (and in turn met with "it's not a bug it's a feature,

And that is the appropriate response to the bug that was filed, with the title "Do not parse "debug" command line parameter".

The bug report makes it immediately clear that this is not really about a technical problem that ought to be solved - no, the dirty user-space programmers should get their stinking paws off of our good kernel command line flags, how dare they!

Kay pointed out in his first comment that there is already a different parameter that turns on debug output only for the kernel; the reporter said he's too lazy to start using it.

And this is not about "entitlement": there is a clear administrative use-case for systemd using the debug parameter: if your system doesn't boot you want a simple and obvious way to turn on debug logging in all of the relevant plumbing code, whether it's in the kernel, in the initrd or during service startup. You shouldn't have to remember five different parameters because then you'll forget one and your outage lasts 10 minutes longer.

If the bug report had been something more like "I've enabled debug and my system failed to boot" without prescribing a particular fix and leaving that at the maintainer's discretion, it wouldn't have been resolved as NOTABUG but fixed quickly.

Of course, a wiser and more socially skilled maintainer than Kay Sievers would have immediately filed a follow-up bug report, or re-titled the bug report instead of lowering himself to the level of the reporter and playing resolve-reopen ping pong. (Fortunately the systemd project also also has some more patient maintainers like Tom and Zbigniew.)

Read the rest of the email you linked, where Linus clarifies that systemd's use of the 'debug' parameter seemed to assume that systemd was the only user of it, thus going against the whole point of a readable /proc/cmdline. Thus, feedback loops were caused and pain was created and flamewars were started.

Again, if you shove your hand into a fire, don't blame the hand (or, for that matter, the fire) for your own arrogance and stupidity. Common bloody sense.

If you discount the usual Linus style hyperbole from the rest of that mail, the only thing that remains is that Linus is unhappy with the fact that Kay closed that bug report instead of trying to find the root cause of the reporter's boot failure.

Honestly I find that a single global "debug" option would have been useful, instead of having to know which component (the kernel, the ramdisk, sysvinit, sysv-rc, systemd, upstart, udev, whatever) is at fault.

Unfortunately such mechanism introduced a feedback loop in the presence of a bug in the kernel. I understand that it would be difficult for the kernel to guard from such loops, but my sincere impression is that they threw out the the baby with the bathwater.

I agree that such a flag would be useful, too, in which case the systemd folks could very well have consulted with the kernel folks in advance to work out a deprecation path for the kernel-only behavior and an introduction path for a generic behavior that activates the respective kernel-specific and systemd-specific flags.

The problem is that such collaboration didn't happen, and the systemd folks assumed that they were the center of the universe and decided "well fuck the kernel, we're going to use the 'debug' flag for ourselves, everything else be damned", thus causing breakage and flamewars. Had they taken the 10 seconds to write up an email on LKML or somesuch saying "hey, we think the 'debug' flag should apply to everything, not just the kernel; y'all game?", the vast majority of the fallout would have never existed.

> the systemd folks assumed that they were the center of the universe and decided "well fuck the kernel, we're going to use the 'debug' flag for ourselves, everything else be damned"

Note that even Linus said that "parsing [the 'debug' and 'quiet' flag] and doing something sane with them is not a bug, it's a feature". So the issue revolved on what people believe is "sane", not about userspace using kernel bootparams (systemd still does and Linus is ok with it). The systemd folks argued that the kernel should not affected if something broke and started throwing too many messages at it during early boot (indeed, the kernel was fixed to rate limit in those cases). The kernel people instead said that it's not ok for userspace to flood the kernel, not even in broken setups (indeed, systemd was fixed to stop sending messages to dmesg as soon as journald is up).

I believe that sending emails here and there to ask "it's ok to do that?" would not work. LKML is already high traffic as is, such messages would just get ignored until things break.

Reply on news.ycombinator.com