Hacker News Reader

> everything that came before it ... sysinit

Your commentary is undermined by the fact that you do not have the experience with the things that did come before it to know that you are talking about the wrong thing. So you cannot be expressing a fully informed opinion. The assumption that only systemd and van Smoorenburg init exist is a fallacy that was called out by the "Uselessd Guy" several years ago.

* https://web.archive.org/web/20190306213420/http://uselessd.d...

The thing that came immediately before systemd, on operating systems like Ubuntu and Fedora, was upstart. It was not van Smoorenburg init.

And that was not systemd's only antecedent. On real Unices there were the Service Access Facility on AT&T System 5 Unix (replacing the running of services through inittab years before Linux even existed), Solaris' SMF, and the AIX SRC. On Linux operating systems there were all of the tools in the daemontools family except for the nosh toolset (which post-dates systemd), InitNG, OpenRC, and a whole bunch of others.

* http://jdebp.uk./FGA/unix-service-access-facility.html

* http://jdebp.uk./FGA/inittab-getty-is-history.html

* https://blog.darknedgy.net/technology/2015/09/05/0/

I agree that systemd gets way more hate than it deserves, but it has continual problems on Arch Linux. Wherever you try to shutdown without logging out first, systemd says ”a stop job is running for user manager...", then waits 90 seconds. The issue keeps getting fixed, then resurfacing. People with this same problem are all over the internet.

In my experience, that happens when a user process is ignoring the SIGTERM signal (ie, a word process or browser wanting user confirmation before shutting down).

Generally, I personally don't mind it and I set the timeout to 30 seconds, which helps.

There is probably no good solution unless systemd starts to send SIGKILL instead of SIGTERM to kill without questioning why, but that would break everything else.

Just to play devil's advocate: prior to systemd, nobody had this problem. In 20 years of Unices I can't remember a single time when I couldn't shut down my machine because a user process was ignoring SIGTERM (or really, because a user process was doing something). I'm not sure what "everything else" there is to break...

I actually enjoy the extra flexibility that I get out of being able to tell systemd how to terminate processes. On the systems I'm building. But it's something that could certainly be improved. It works right for my use case (embedded systems) but I hate it when it does it on my laptop or my home machine.

The whole interface around it is silly. It tells you that a job is running, but it doesn't tell you what that job is, what that process is, and unless you enable the debug shell, there's no way to check it or kill that process either. All you get to is stare at the thing counting down from 90, and in some cases when it reaches 0, it starts counting down again instead of rebooting.

Well, before it wasn't a problem, largely because the init system just murdered everything that didn't drop dead when told to (ie, your desktop apps will be SIGTERM'd along with the X Server). Systemd certainly takes a more safe approach.

Atleast in my experience, systemd certainly tells you what is hanging. The issue with "timer reset" is that some service that is hanging is also blocking some other service from shutting down (either because the hanging service depends on it or the blocked service needs to be shut down after the hanging service). Old Init systems didn't care either, if the system was to shut down, it just killed in order of runlevel so stuff like "The database was corrupted because init forcibly killed the NFS mount too early" was certainly possible.

Edit: first and foremost -- sorry, I went on a trip down the memory lane because you mentioned NFS and I forgot to mention the most important point.

That message, and this approach, is 100% inconsequential, and it doesn't make anything safer. IIRC, last time I checked, virtually none of the distributions enable the systemd debug shell. If you see that message, you can't stop the application manually. Ctrl-Alt-Del doesn't work, either, and if systemd can't recover from that state -- which, depending on what's actually happening and on the configuration, it sometimes can't -- all you get to do is press the reset button anyway.

If an application is hung for a legitimate reason, like, there's unsaved work, you can't go back and save it anyway. All you can do is wait for 30 seconds for the system to reboot and lose the data anyway (just 30 seconds later).

That aside, I've debugged my fair share of corruption, back when journaling filesystems were new and NFS was way worse than it is today. I don't want to go back to those days, either, and I'm glad to have a safety switch for sensitive setups, or for misbehaving/legacy applications.

But no one has a problem with that. People have a problem when they don't have anything mounted over NFS (that, by the way, is also something that occasionally caused systemd to hang at one point) and still can't reboot their machine.

It's not like we had to wait until 2010 to figure out that init systems need to handle these things carefully. Nobody compares this behaviour with the golden age of Ultrix and says oh, right, thank God for this, it's a minor inconvenience but at least my files are safe. Everybody compares this with what's available on every other platform that's popular today, not in 1996 -- namely a "Force reboot" button that reboots your machine and virtually never results in data loss.

> it doesn't make anything safer

Why does a program given SIGTERM not quit? Is it because 1) it's still busy desperately trying to save something large and unusual or over a network, 2) it's antisocial and ignores SIGTERM, or 3) it's hung for some permanent reason?

It's hard to tell from the outside.

Most of the time it's the second or third, but the behavior occasionally saves data in the first case.

I mean... yes, in theory that would be the case but that's really not what's happening in this particular case.

(Edit: Firstest of all, actually, more often than not it's actually 4) the program is waiting for a user's confirmation, or requires some user action to exit, but if you see that message, you can't do it anymore.)

First of all, not being able to distinguish between these three cases is precisely what makes virtually everyone who's seen this a couple of time hit the reset button as soon as they see it. That doesn't save any data.

Second, if a program is really busy desperately trying to save something large and unusual over a network and you see that message, there's a good chance it won't be able to save anything anyway. The network is usually down by that time, for example.

There's also no way for a program to signal that it's done. If, indeed, a program needed one more second to write everything, it's cool, but you still get to wait 29 more seconds.

And finally, processes ignoring SIGTERM is really just one of the reasons why this happens.

I think all of us in the FOSS community should be more charitable when discussing these things. systemd is a big step ahead but it's not like we had to wait until 2010 to figure out that init systems have to tread carefully during shutdown to avoid data corruption.

Also, people who read HN usually understand that killing processes can cause data corruption under some circumstances. Just sayin'.

>First of all, not being able to distinguish between these three cases is precisely what makes virtually everyone who's seen this a couple of time hit the reset button as soon as they see it. That doesn't save any data.

It saves data if you don't mash the reset button at first sight

>Second, if a program is really busy desperately trying to save something large and unusual over a network and you see that message, there's a good chance it won't be able to save anything anyway. The network is usually down by that time, for example.

Systemd won't bring down the network until all services that depends on the network have been stopped. The desktop environment doesn't unless the home directory is a NFS mount.

>There's also no way for a program to signal that it's done. If, indeed, a program needed one more second to write everything, it's cool, but you still get to wait 29 more seconds.

A program running as a service can override the timeout in it's unit file.

>And finally, processes ignoring SIGTERM is really just one of the reasons why this happens.

Correct, processes who ignore SIGTERM are the real problem, processes that take a long time are a secondary culprit. But systemd can't do much if the devs of software refuse to read documentation on signals when writing handlers for them.

i think the source of the hate is how little the dialog actually tells the user. imagine the message saying something like "{{ process_name || process_command }} is blocking the shutdown, press F7 to terminate it forcefully"

i doubt anyone would complain about systemd with such a message. its just beyond annoying not being able to see which application is causing the issue nor having any options beyond waiting or removing the power entirely

I get this every time I shut down. The GUI has shutdown a while before, but I have to hold the power button down every time I shut down my laptop.

TBH it's not as bad, as suspend being broken because it's a ryzen based laptop. (Half the time it comes back with a black screen and I have to hard reboot).

The GUI shuts down but the process of the app usually keeps running. It's not unusual that I see firefox being waited on.

If he's seeing that message, he's already at the console which means that the X server has been shut down.

Plenty of apps will keep running, atleast from what I see, the console is up and usually waiting for firefox to close (which is waiting for the confirmation dialog).

It should register itself as an inhibitor then. Systemd won't let you shut down the system as long as there are inhibitors active (you can still force it). As desktop Linux users we should consider it to be a bug in Firefox (among others). See

     $ systemd-inhibit --list

https://www.freedesktop.org/wiki/Software/systemd/inhibit/

Cue "applications-shouldn't-depend-on-systemd-we-need-350-competing-init-systems" folk.

It'll probably take some time for desktop apps to register as inhibitor. KeepassXC has done this IIRC so now if systemd tries to shutdown with an unsaved database, you can't. Various other desktop apps do too these days, so it's largely a question of time.

What do you mean, exactly? Does the X server stay running after you're already looking at the console? Or does Firefox stay running after the X server is already gone?

Firefox keeps running after X is gone and waits for me to confirm to close all windows.

Maybe there should be hard and soft shutdowns? Or maybe there already is?

SysRq-b is a hard shutdown. "Soft" vs "Hard" isn't a realistic distinction, systemd already does a "hard" shutdown if the timeout is exceeded. There is no reason to do a hard one in any system, a soft shutdown is always preferable as things like database engines might need a moment to save their data. If you send them a SIGKILL then your database engine will loose data.

Back in 2017 there was a person on Twitter who was collecting every time that one of his machines had a systemd shutdown problem.

* https://news.ycombinator.com/item?id=15896376

Machine shutdown is one of the parts of service management that often does not get enough attention.

People shutdown their computers?

This is arch. You're likely getting at least one kernel update a month. That requires a restart which involves a shutdown. (Unless you're into kexec magic)

I have a number of systems running with systemd and a number running with other inits (runit or shepherd). I continue to have more problems with the former than the latter two.

> Timers are generally better than cron.

(which cron? Many cron daemons are available. While the traditional Vixie cron was popular with distros for historical reasons, it wasn't difficult to switch to another cron variant or any other job scheduler)

If that feature or other components like journald were designed as an independent package instead of being unnecessarily coupled into systemd, we could all simply choose which tools we want to use.

That's really a bad example. Timers are just standard units that are executed at regular intervals instead of at startup/when a specific file is requested/when another service is started/etc.

It makes perfect sense for this particular feature to be part of the core capabilities of systemd units.

Journald would be a better example but FYI most systemd components are optional and only a few of them are required. Whether it makes sense for those components to be required is up for debate and there are good arguments on both sides.

The systemd team decided it made sense to have a hard dependency on those components. You can disagree with them but the only alternatives you have is to either fork, come up with your own init system or join the core systemd team and try to make them change their mind from within (and probably put up the hard work needed to remove the hard dependency on those components).

People seem very passionate about init systems since systemd was introduced but few seem passionate enough to do the hard work needed to implement the changes they would like to see.

> The systemd team decided

No they did not. Nominally, at least.

The systemd people have always maintained that this stuff is up to the people who package systemd for individual operating systems. It is those latter who decided that (a) there would be one big systemd package and (b) other packages would have dependences from it. They could have decided to package systemd up as multiple packages, instead.

In fact, a lot of the Debian Hoo-Hah was over the packaging.

* http://jdebp.uk./FGA/debian-systemd-packaging-hoo-hah.html

This is slightly undermined by the fact that a few of the prominent systemd people are (or have been) the systemd package maintainers for various operating systems, so this is in fact in some cases the same people simply with different hats on their heads.

It's a great improvement.

As someone that has gone through Redhat, SuSE, Debian and Ubuntu over the years having something that works (efficiently) on all these platforms is great.

Trying to work out the init scripts in some distro when you are used to another is a fun way to spend an evening, I don't have the free time for that any more.

It is functionally better but far more brittle, over complicated, heavily coupled, stateful and poorly managed.

The net gain is close to zero. Somewhere between sysv and systemd was the sweet spot.

Edit: I know my point is unpopular, perhaps mostly with RH staff browsing HN late at night, but this is experience from running systemd in production over 4 years on CentOS 7. What was not a concern before is now a concern and is actually costing me time, money and friction regularly. Add the Linux boot process, all the FreeDesktop crud to this and it's a nightmare of a system to look after. Windows/FreeBSD/OpenBSD are far far far easier to manage from this perspective.

> brittle and poorly managed

Could you expand on that?

Brittle - try dealing with a hosed boot or DBus failure. I've dealt with numerous on CentOS 7 and at this point it's easier just to blow the whole node away and start again. Also I've had a couple of segfaults on journalctl recently when I actually need it so i have to then go digging through the GUID encrusted hell it leaves on disk.

Poorly managed - Pwnie award, "it's not a bug" from Poettering etc etc.