Hacker News Reader

Comment by otterley | Hacker News Reader

How, exactly, has systemd "broken user space"? That's clear neither from the lkml thread, nor here. I'm not sure what this discussion even has to do with systemd; and I'm not even sure how you can "break user space" from a user space program, which init is.

You might be able to modify your descendants' environment, but the rules under which that can happen are maintained and enforced by the kernel. It seems to me that if user space programs can do undesirable things, that's the fault of the kernel itself. To be sure, setting those boundaries and rules is one of the kernel's main jobs.

View on HN

Another example:

    $ nohup ./long_running_process &
    $ logout

Previously: long running process continues running after you logout. Recent change in systemd: Process is reaped once your login session ends (depending on flags passed via ./configure when systemd was compiled, your distro of choice might not be affected right now).

You could interpret this as "breaking user-space", as it's contrary to long-running practice and user-expectations.

Or you could say that systemd is superior as it's reliably cleaning up unwanted lingering processes which may have bothered some before. And this is worth the hassle of explicitly running intended background processes via systemd-run(1), or adding code to e.g. tmux/screen doing the equivalent via the dbus-api.

I actually explained in detail how one could fix this, a year ago.

* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394#221

Absolutely. I cannot fathom why they thought that SIGKILL was appropriate instead of SIGHUP, even if they made the misused choice of considering it their responsibility at all.

Dagnabbit systemd! Thank you. You just solved a mysterious bug for me.

The whole purpose of nohup is to get your process to ignore your session ending. Why on earth systemd would up and decide that you didn't do that on purpose is beyond me.

When people say they hate systemd, it's precisely this kind of thing that we're talking about.

Yes, this breaks something that has worked in Unix for 30+ years. Awful.

Systemd does give us extremely quick boot times, though! Too bad I generally don't reboot my Linux systems more than once a month.

Indeed. systemd does boast respectable boot times (not as fast as hand-crafted shell script, but pretty good for a generic init).

On the other hand, error handling is pretty awful, especially during shutdown: Every now and then systemd would decide to give that one unresponsive sshd process a generous couple of minutes to shut down. Or will wait for a random filesystem to unmount itself for another 90 seconds. Sometimes outgoing systemd will lock up completely, until you do the magic ritual of pressing ctrl+alt+del 7 times within 2 seconds all while spinning around on your toes trice counter-clockwise and barking - to force "immediate" restart. Better yet - on special occasions this ritual devolves into farce: _after_ hammering ctrl+alt+del at machine gun rate it will print "rebooting now" and... lock up again. Until you hammer ctrl+alt+del again - only to see another empty "rebooting now" promise. SysRq would - of course - have no problem at all remounting filesystems ro and rebooting from this sad limbo.

The more one read the more one get flashbacks to the very Windows painpoints that made one adopt Linux in the first place...

Here's a good explanation of the behavior, how to disable it, and why Fedora will be turning it back on in 26 (it's been off by default). https://fedoraproject.org/wiki/Changes/KillUserProcesses_by_...

I don't get it. They say that the default should be to kill everything that doesn't opt out, but that already has been the default. SIGHUP gets sent to all processes, and those processes can explicitly ignore SIGHUP. In the new scheme, SIGKILL gets sent to all processes, except if the process has requested not to be killed through systemd's API. A misbehaving program in the old scheme would still be misbehaving in the new scheme, so there is no added benefit.

The added "benefit" is that once more systemd gets to define behavior.

Systemd is not about building refinements on unix, it is about turning Linux into something like OSX or Android. a OS that use unix semantics only for bootstrapping convenience.

Effectively they are turning the Linux kernel into a convenient source of drivers, but beyond that could not care one bit about unix.

I really wish i could find the interview again where it mentions that Poettering in the past have advocated throwing away the chapters on unix programming from one of the seminal works on programming unix and Linux.

That's what I'm getting, but I'm trying to at least give systemd advocates the benefit of the doubt. When I have the time and energy, it can be a good use of time to poke at people with questions.

No, that's just Zbigniew Jędrzejewski-Szmek's explanation repeated ... by Zbigniew Jędrzejewski-Szmek. I see from elsewhere on this page that you have unwisely bought into the idea promulgated on that page that Zbigniew Jędrzejewski-Szmek will "work with upstream authors and Fedora maintainers of programs like screen and tmux". No, xe won't; and demonstrably has not.

You need to look at some of the history here. Over the course of the past six years, Zbigniew Jędrzejewski-Szmek has twice cycled around the same loop, once in 2011 and once in 2016. Xe goes to the developers of tmux, tells them that tmux needs to change to be systemd specific, and the developers of tmux ask why systemd cannot provide the same semantics for interactive login sessions that have been employed for the past 38 years, with HUP for session hangup and TERM/KILL at system shutdown; or why at the very least xe does not talk to the C library people. Zbigniew Jędrzejewski-Szmek goes away, and then comes back about 5 years later with the exact same thing, unchanged.

Unfortunately, buying into that Fedora wiki page is as unwise as buying into the rather erroneous Freedesktop explanation of how version 2 cgroups work. You will note that the wiki page dates from around the same time as Zbigniew Jędrzejewski-Szmek's last cycle around the loop, which hit the headlines last year.

* https://news.ycombinator.com/item?id=11797075

* https://news.ycombinator.com/item?id=11797926

* https://news.ycombinator.com/item?id=11782364

Unrelated, but how come you've chosen to use the `xe` pronoun?

I couldn't find any mention of it in Zbigniew's blog.

FWIW, "Zbigniew" is a male forename.

From what I can tell, they use it for everyone.

[0] https://news.ycombinator.com/item?id=11785977

[1] https://news.ycombinator.com/item?id=14665518

[1] https://news.ycombinator.com/item?id=14580113

[2] https://news.ycombinator.com/item?id=14553959

I've sometimes used it (or ve) in the past, because there's no good gender-neutral pronoun and I don't like to assume. (This is partly a politeness thing and partly an accuracy thing.)

Occasionally when I would do so, people would get confrontational about it, so I mostly stopped. But now that means I sometimes waste time trying to find someone's gender even though that's totally irrelevant, and/or try to work around needing a pronoun at all, and/or use "they" even though I don't like it. None of these solutions is great.

(Btw, I don't parse you as confrontational. Comments like yours wouldn't have bothered me.)

There are a couple of inits in common usage on Linux; systemd is now the most popular and ships standard on the most popular distributions. I can't imagine Linus would be feeling grouchy toward a nearly defunct init (that had been in use mostly unchanged since the 90s), so we're left with systemd. Unless you've got a better idea about what he means?

It is not a secret that kernel devs have a love-hate relationship with systemd, and have for years. That's why Linus said, "You all presumably know why." He was speaking with context that everyone on the LKML already knows.

First result when I google it (but it's been a thing for a long time and on many fronts with many interesting bugs and arguments): http://www.phoronix.com/scan.php?page=news_item&px=MTY1MzA

I don't really subscribe to the "systemd is terrible and is the worst thing that ever happened to Linux" theory, and I'm not trying to be one of those folks that takes every opportunity to make a thread about how much systemd sucks. I like systemd. I mean it when I say that. But, in this case, it seems obvious it is about systemd. I have a hard time coming up with another plausible theory.

Edit: Also, now that I'm thinking about Linus and Poettering butting heads...well, that's hilarious. A couple of brilliant assholes arguing is funny to me. I dunno how much it's actually happened (and how much happened through proxies), but still. Funny.

But you said specifically "...and systemd has broken userspace a lot."

What did you mean by this? It seems like a complete nonsense statement in the context of user space breakage of the form the kernel is responsible for.

The main clash between the kernel maintainers and the systemd maintainers was over systemd reading the kernel argv in an error prone way to trigger debug information. Other than that the interactions seem completely minimal. Linus Torvalds uses systemd at work and at home.

"What did you mean by this? It seems like a complete nonsense statement in the context of user space breakage of the form the kernel is responsible for."

Does it? systemd is the init, the service manager, the interface to D-bus (and kernel devs had a bit of a tussle with the systemd devs about kdbus), the logging interface of most services (possibly including kernel logs), etc. systemd interacts with both the kernel directly (as the system init, as one interface to cgroups and namespaces, etc.) and with users (as everything else it does). I like it, but I understand the Borg accusations.

"The main clash between the kernel maintainers and the systemd maintainers was over systemd reading the kernel argv in an error prone way to trigger debug information."

Searching the LKML for systemd seems to indicate otherwise. It seems like there have been a few scuffles over the years. I don't subscribe to the LKML or read it religiously and haven't for many years, so I'm mostly guessing based on context, but the context seems clear to me.

Again, I ask, if not systemd, then which init is Linus talking about?

(Disclaimer: I hate systemd)

The issue isn't him not knowing what systemd is, or not knowing that that is the init system Linus is talking about. He is aware of both.

His issue is with your claim that systemd has broken userspace. You haven't seemed to provide any evidence of this being the case.

We may simply be using different terminology. systemd has had bugs (a lot of them), many of which impacted the system at a high level (being the init and everything else), sometimes due to ineractions with the kernel, sometimes due to interactions with other parts of the system. Perhaps others wouldn't call that "userspace", I dunno. I figure if it affects users trying to run software it's "userspace".

But, I'm happy to take the "broke userspace" assertion back if you don't like the term. Let's just say systemd has had a lot of bugs, sometimes stupid ones, and sometimes stubbornly held on to despite kernel devs or other devs asking for changes. I would hope we're not arguing about whether systemd has had an exciting number of bugs and compatibility issues (I think we can all agree on that, even if we like systemd).

I inferred a bunch from my recollection of how the LKML has talked about systemd in the past. I assumed Linus was talking about systemd and that when he says he doesn't trust it, it was about systemd being sloppy and stubborn about compatibility and fixing bugs. I could be wrong about that.

Linus is generally very specific when he talks about things breaking userspace - changes to the kernel should not make things that work in userspace stop working.

systemd doesn't really have the power to completely break userspace in the manner Linus refers to, though it certainly has the power to break a system.

(One of my favorites was when a systemd update shut down dhclient but did not restart it, so once dhcp leases expired, servers lost connectivity. It was fun seeing several thousand machines go offline all at once. "Fun.")

But Linus wasn't the one who wrote about breaking userspace here, so Linus' use of that term is irrelevant.

It's a term of art -- one defined by Linus and the kernel team over the past 25+ years. People who have been involved in one way or another with kernel development know what it means, and it makes little sense to redefine it here.

English is defined by usage, and whether or not it make "little sense" to you, if people use it differently and makes it clear they mean it differently, it makes even less sense to try to belabour the point based on a different definition.

Your point is taken that English is indeed an evolving language, but there's a strong benefit to consistency that is illustrated here.

If definitions are inconsistent in the minds of people participating in a conversation, it can muddy an issue and make people needlessly argue because they aren't communicating effectively about what they mean. It also makes it more difficult for a reader to understand the debate.

Specifically, in this discussion, when I said "breaks userspace," I and others were referring to the term of art as developed over the past 25 years, and I expected (this being HN) that others would also understand it as such and discuss accordingly. Instead, this thread has turned into a morass of misunderstanding and confusion, because still others believe the term means something else, and are arguing a tangential point as a result.

You're right. I should be more careful about language. When talking about kernel developers, they do usually mean the ABI.

Userspace programs that others frequently depend on the behavior of can easily break userspace. Look how many things break when you change ps or ls out with another tool.

Maybe eventually Linus will become so fed up with systemd, that he would write an init system of his own, which would become a major success just like git did. Then conversations like this one will be pointless, because everyone and their dog with the possible exception of RedHat would have long switched to Linus' init.

Systemd and the whole controversy around it is anything but funny. It's upsetting and it already divided the community.

It's a little funny.

systemd has some great stuff in it. I have my nits to pick with it but, on the whole, it's better than what we had before. And, don't we all just want things to be a little better every day?

I think it's all gonna work out fine in the end (and if not, we'll all be dead someday, anyway).

I would say the same with pulseaudio; before this, there were a mess of audio servers (esd / ksd or none) - now things are definitely better.

I have not required a audio server in years, and i have not adopted PA. What i did was enable dmix, that comes with Alsa.

A behavior that is even default Alsa these days if it detect that you use sound hardware without mixing.

The one "issue" that PA "fixed" was that of temporary audio devices (bluetooth, USB). But then i find the whole notion of USB headphones (a pair of headphones soldered to a minimal USB sound card) a massive abomination. and i fail to see why paired Bluetooth audio devices are exposed directly into the /dev tree rather than behind the Bluetooth dongle device.

I use tmux and emacs --daemon constantly. My experience with systemd is that I need to check whether KillUserProcesses is enabled, or else systemd will kill my processes. My experience with init was nonexistent, because it never came up as an issue.

Most distros have set KillUserProcesss to no for the short-term. Fedora plans to make all of the obvious user programs aware of the change (and able to create a new systemd unit for the process) and turn it back on for Fedora 26.

https://fedoraproject.org/wiki/Changes/KillUserProcesses_by_...

Their reasoning is pretty good, IMHO. It provides more control at low cost, and it's a one-line configuration change if you want the old behavior. But, everything I need it for (pretty much just screen and tmux) will already be aware and will handle setting up the new systemd unit automatically.

But, it breaks a standard Unix behaviour going back years, and it can't be configured by a non root user.

It is a one-line configuration change IF you know that it exists, and you have administrator privileges on the machine. If either of those conditions fail, systemd has broken your setup.

Not only that, but there was an existing mechanism for this. On a disconnect, SIGHUP would be sent to all processes. A background process can explicitly ignore SIGHUP in order to remain running on disconnect. I would be fine if systemd were sending SIGHUP to processes, because that would fit with the existing method of opting out. Sending SIGKILL, and having to use systemd's API to opt out of being killed is ridiculous.

This adds more complexity for no added benefit. My standard .bashrc now needs to have a line checking whether KillUserProcesses is enabled, so that I can bug the admin to disable that idiocy if it is.

>A background process can explicitly ignore SIGHUP

The problem with SIGHUP is that a background process can't explicitly ignore SIGHUP, it can only implicitly ignore it. If you send SIGHUP to a process and it doesn't exit you have no idea if it didn't exit because it wants to hang around or if it didn't exit because the process is hung.

Part of the motivation for fixing that is to eliminate things like gnome-keyring-daemon hanging around after the user logs out.

Correct me if I am wrong, but I would call signal(SIGHUP, mysignalhandler) to be an explicit ignoring of SIGHUP. I have taken a non-default action in order to have a non-default behavior.

Since gnome is the one that requires behavior and integration with systemd beyond what can be done with signals, a reasonable workaround would be to have the extra logic in systemd apply only to gnome. As it is, they are changing to a default that is entirely unreasonable outside of a desktop environment, and requiring others to work with the new system.

>I would call signal(SIGHUP, mysignalhandler) to be an explicit ignoring of SIGHUP. But that doesn't explicitly ignore SIGHUP. You're explicitly handling it but from an outside perspective we don't know if your SIGHUP handler is supposed to be a no op, output some progress information, terminate the program in some special way or what. All the system knows is that it signaled your program and your program is still around 10 seconds later. In terms of your program, it hasn't taken any external action, hence not being explicit.

As for the Gnome project though, that's just an example. I'm talking about cases where the application should terminate upon receipt of SIGHUP but doesn't. That default isn't changing, the method to avoid the default action is. Back when SIGHUP was created all software that wanted to ignore it needed to make a handler to avoid the new default behavior. If you want to be able to kill misbehaving processes that were supposed to terminate on SIGHUP then the design needs to change. There's no way to fix this without making some breaking change.

As to requiring others to work with it, your distribution is the one that's making you do that. systemd upstream has had killuserprocesses for a while now. It's just a configure option when building systemd. It's not like your distro is having someone else build packages for them, setting appropriate config options is entirely on them. Some distros are building systemd with that on by default, and some aren't.

So, if I understand correctly, a misbehaved program that incorrectly ignores SIGHUP will be replaced by a misbehaved program that starts a new user scope with systemd. In both cases, there is no way to tell whether the program has hung, or whether it is working as intended. The end result is the same. So, we've switched to a new API, broken all existing daemons, for no end benefit?

I'm fine with having KillUserProcesses exist as a concept. There are some odd machines, like public terminals, where you would want to forbid any long-lived processes. I'm not okay with it being the default option. Yes, the distributions can change their default, but systemd's default in an endorsement that shouldn't be there.

No, that's not the concern here. KillUserProcesses is intended to reap children that were never intended to persist outside of the user session. My ssh-agent is never going to move itself into a new scope, but it very well could have a bug that causes it to hang around after I log out.

The only things that will ever create a new scope are things like tmux, screen, and maybe some variant of nohup.

Here's the kinds of problems that misbehaving processes actually cause.

https://bugs.freedesktop.org/show_bug.cgi?id=94508#c10

And yes, KillUserProcesses reliably fixes crap like "hp-systray" from hanging your whole session in a way that isn't obvious to the user. If there's a deadlock when trying to log out, somethings gotta give otherwise your session will just persist forever.

> Back when SIGHUP was created all software that wanted to ignore it needed to make a handler

False. People ran such softwares via the nohup command. No program modifications were needed. They still aren't.

Yeah but that's exactly the same situation we're in right now with distros turning on KillUserProcesses. I can start whatever processes I want to in another scope without having to modify those processes.

... but not with general-purpose tools that use the standard semantics that have been around since 7th Edition. Only with Linux-specific and systemd-specific tools. Which is why it of course is not the same situation at all.

Maybe eventually Linus will become so fed up with systemd, that he would write an init system of his own

Can't. Fucking. Wait. Seriously, that day cannot come soon enough. systemd might have some useful ideas, but their implementation is crap, and the overall intrusion on how to get stuff done is significant.

> a couple of brilliant assholes

Define brilliant. One changed the computing landscape by starting (and running for decades) the biggest opensource copyleft project in the history of mankind; the other is a bog-standard corporate engineer who simply leverages his parent company status to piss off most people and get away with it. Placing them on the same level is an insult to Linus, who might occasionally be an asshole but is still the most effective leader the opensource community has ever seen.

Assholes can be effective leaders of community.

I think the only problem that the parent comment had with the quote was about the brilliant part being assigned to Poettering.

EDIT/NOTE: I don't have opinions on either Linus or Lennart. I use both of their software daily.

The funny thing is that Torvalds is not much of an asshole unless you violate the trust he have placed in you.

He will gladly chastise himself for errors etc, and is quite humble in public.

Poettering on the other hand have a history of hubris and assholeness.

Consider for example that he not only heckled a presenter at a conference. But when the presenter bowed out early he climbed the stage, beer bottle in hand, to grab the microphone.

It would not surprise me of Poettering have never met hardship during his school years (apparently the signing scheme used in journald was based on his brother's doctoral thesis no less) until he hit the net with his projects, and simply do not know how the handle negative feedback.

Poettering is perhaps an asshole, but Torvalds reputation for being so is overhyped to put it mildly.

Never mind that they have faced off (sort of) on camera at least once, with Poettering basically missing the point again and again that not just Torvalds but also other long term kernel devs on the stage was raising.

systemd changed the way it worked with autofs, and broke userspace. They added a workaround, rather than trying to work with how the kernel already worked. (Which was highly controversial at the time.)[0]

And the systemd opinion on breaking userspace[1] is that compatibility doesn't need to be maintained:

>> The kernel's policy is "don't break userspace" - isn't the init(1) equivalent "don't break the rest of userspace"?

> No it isn't. We need to break things where progress in the base OS is more worth than compatibility.

[0] https://lists.freedesktop.org/archives/systemd-devel/2011-Se...

[1] https://lists.freedesktop.org/archives/systemd-devel/2014-Ja...

> How, exactly, has systemd "broken user space"?

userspace was working, systemd got an update, userspace now not working.

systemd hasn't broken userspace, but it regularly breaks programs that make the (bad) decision to use it's D-Bus APIs. If you've ever tried to interact with systemd's TransientUnits you know exactly what I mean.

runc has been broken many, many, many times by systemd.

So when my computer fails to boot following a system update due a change in how fstab is handled, isn't that a broken userspace ?

It's a broken system. I never said systemd doesn't break systems. "Breaking userspace" has a very specific meaning in this context, and it's when a kernel API changes such that an existing program no longer works on the new API. systemd is not part of the kernel so it doesn't dictate kernel APIs.

I agree with your dislike of systemd, I disagree with the term you used.

By your argument, whenever any program used by anything else has a bug, that's breaking userspace -- which defeats the "userspace" qualifier if every breakage is breaking userspace.

Reply on news.ycombinator.com