Systemd, ten years later: a historical and technical retrospective
blog.darknedgy.net
blog.darknedgy.net
What’s much harder is (1) to expand ones code without coupling everything together (2) present it in a persuasive way that naturally builds a user base.
SystemD feels like the new grad hire who decided the first thing the would do after joining — the opening power play — is to convince the PHB we should rewrite everything in Haskell.
At this point the gentler folk start quietly honing their CVs and sneaking their personal belongings off their desks, one night at a time, in the hope no one notices they are all about to quit.
It felt like Unix systems administration was a haven from this sort of politics — I don’t remember tmux, git, or python trying to push behavior changes on upstream components of the OS[1]. I’m sure it’s why Unix attracted a certain type of personality for so many years and I’m sad to see that changing.
[1] If you are thinking “but those are command line utilities, whereas the init process is a much more specialized case” then I’d encourage you to revisit the Unix principles of every component being a small and simple program! Even init, cron, and login! It’s not some grand ideal to be zealously adhered to: it’s the principles of small components that mean the same OS can run on a quad core Xeon as runs on a $45 network switch.
It was pretty good in the good, old days, where a sysadmin could list every daemon the system runs by heart, but it was breaking apart when you had tons of random packages:
Package X uses "start-stop-daemon" in ini file, so there is no way to get error messages if config file is incorrect. Package B has no disable switch, so you get to stick "exit" in /etc/default. Package C has this weird logging format which cannot be rotated daily, and is not integrated with anything. Package D likes to die and leave PID file around, and then it fails to restart because of that. Package E likes to bury its settings in /etc/init.d/ file, so there are conflicts every time it updates. User A likes to leave their executables on their machine after logout, so we need a cleanup script.
This was seriously getting old -- after you apply a hack after hack, you start wanting something better. There were a few candidates -- upstart, daemontools/runit, systemd. Only two of them actually did the work to add backward compatibility and integrate with upstream. Upstart sucked even worse than systemd. This left only one candidate.
Remember, even if you can claim for Fedora that someone "convinced the PHB", this does not apply to other distros. Debian, Ubuntu, Arch all chose the systemd voluntarily. This is because sysvinit sucks, and no one came up with a better option.
It's as though someone is complaining about having to live under the USSR and how bad things are, and then someone responds that the German rocket program has been dismantled and the British never really had much of one and the Americans are doing okay but the fact is that the USSR was the first to put a man into space. As if that justifies everything else they did.
What you could do is to reduce unnecessary interdependencies. And where there is inherently interaction between separate packages, use a stable interface and publish the standard so that alternative implementations can replace a single package in a standard way without breaking several others.
I'm talking about all the things that now break not related to the normal function of init at all.
If your complaint is that the scope of init systems has been expanded, I don't know what to tell you other than that it was inevitable. Systemd is not the first project to try and do this.
If the "competing alternatives" don't need it then I would advise those projects to put their money where their mouth is and either start making improvements to elogind, or develop a new login manager that better fulfills the needs of gnome. In particular the solution used by elogind to deal with cgroups is to just disable them entirely, or at least it was last time I checked. I don't know what runit does for cgroups but a good step would be to make elogind compatible with that.
elogind is a stub piece of systemd and systemd refuses to make any commitment to maintain compatibility. It's like saying that windows programs work on Linux, you just have to use wine - it's sort of true for now, but it's not something you can rely on.
> And, there were valid technical reasons to have gnome depend on a login manager, in particular supporting multi-seat securely is very difficult without one. I should also mention that the previous "standardised interface" for this was ConsoleKit which was also written by the same developers as gnome and systemd.
I remember multi-seat working fine long before either systemd or ConsoleKit, so that seems pretty questionable.
> I would advise those projects to put their money where their mouth is and either start making improvements to elogind, or develop a new login manager that better fulfills the needs of gnome.
"The needs of gnome" are a constantly moving set of goalposts under the control of RedHat. Gnome worked fine long before systemd and systemd has not noticeably improved it (if anything the opposite); there was no technical consensus that the current hard-dependency was necessary, it was forced through because RedHat wanted it that way. So they're not going to accept patches to offer compatibility with non-systemd, and if they do then they'll just find a different way to hard-depend on a different part of systemd.
Isn't that the whole reason for using another init? Because you see full compatibility with systemd as being undesirable and you have the means yourself to maintain things yourself using other solutions? I don't get it, it sounds to me like you're now saying the opposite of what you did before.
>I remember multi-seat working fine long before either systemd or ConsoleKit, so that seems pretty questionable.
Multi-seat might have worked but it did not do so securely. The issue is that you need a daemon to multiplex the display and input devices, and programs that want to request logins must be able to speak a specialized IPC protocol to that daemon. Otherwise all you have is a bunch of setuid root programs that do not communicate and can easily clobber each other at any time. Remember that the kernel does not have any built-in support for this, these are just device nodes in /dev like anything else, and in the old days the X server just used to run as setuid root at all times and sessions running in the background could read the keyboard whenever they wanted or take control of the display. The issue doesn't just affect multi-seat either. Even on your laptop you basically need a login manager to do secure screen-locking. There are an absurd number of ways that pure X- or Wayland-based screenlockers can be escaped. In an ideal world the kernel would probably prevent this stuff, but it doesn't right now, so login managers remain necessary.
>"The needs of gnome" are a constantly moving set of goalposts under the control of RedHat.
1. GNOME is an open source project not in control of any one company or organization, and anyone can contribute or fork the project if they so desire.
2. Nothing was "forced through" as the decision was agreed upon by the current and previous maintainers. If you disagree, you can revert to an old version as previously stated. This may be unpalatable to you but it is not realistically possible for anyone besides you to write code that you will agree with 100% of the time. It's up to you to make the tradeoffs you want on your system.
3. Also as previously stated, there is no hard-dependency on systemd. You can get it working with elogind. If you refuse to because you fear it will break in the future, that's on you. The fact is that it works now.
Compatibility is the only way it becomes possible to make those changes. E.g. in the pre-systemd days there were at least four different syslog systems and at least three different cron systems. But because they conformed to standard interfaces and were loosely coupled to the rest of the system, users were free to choose any of them, or to make their own. Same with desktop environments: you can run KDE or Gnome or Xmonad or whatever, but since they all follow common standards, any program works with any environment.
If the systemd people want to make a shiny new init with cool features, great. It's when they break compatibility with alternatives that I have a problem with it.
> 1. GNOME is an open source project not in control of any one company or organization, and anyone can contribute or fork the project if they so desire.
> 2. Nothing was "forced through" as the decision was agreed upon by the current and previous maintainers. If you disagree, you can revert to an old version as previously stated. This may be unpalatable to you but it is not realistically possible for anyone besides you to write code that you will agree with 100% of the time. It's up to you to make the tradeoffs you want on your system.
The fact is that most contributions are made by RedHat employees on RedHat time, so anything that can be decided by 51% of maintainers is under RedHat's control (and I can't do anything to change that since I can't outspend RedHat). There was a decision to adopt systemd but it was very much not a consensus; there was bitter disagreement at the time (and still is).
Forking by a minority is only practical if the system is made up of small components with standardised interfaces between them. Good open-source citizens follow those standards for the sake of allowing their users to do that (e.g. KDE migrated from DCOP to the technically inferior DBUS for the sake of being able to have common standards and compatibility with GNOME). If the whole system is a big ball of mud with no way to replace or decouple individual pieces, then you can't ever change or fix one part of it without forking the whole thing, and you don't have the Four Freedoms in any practical sense.
That is not relevant because the "common standard" in this case is the logind D-Bus API. There are no other active and viable alternatives to login management. It has two implementations, systemd-logind and elogind. You can choose either one, and only the first one requires systemd. Or if you really want you can choose to write another implementation. Or you can choose to hack GNOME and remove the logind bits and just run your display manager and shell insecurely as setuid root. Or you can revert it to some old version when consolekit was still supported. Or you can just do nothing and wait for someone else to guess what it is you want. You have options. If none of these will please you then I think you need to adjust your expectations because you can't have it all.
>The fact is that most contributions are made by RedHat employees on RedHat time, so anything that can be decided by 51% of maintainers is under RedHat's control (and I can't do anything to change that since I can't outspend RedHat).
I don't see what this has to do with anything. It's open source. If you want to collaborate, you can submit a pull request, work with the maintainers and get them to spend their paid time reviewing and merging your code. You don't need to outspend them, the code is already written and every bit of it lands on github ready for you to use it. Them getting paid more only benefits you.
>There was a decision to adopt systemd but it was very much not a consensus; there was bitter disagreement at the time (and still is).
I can't speak for other distros but looking at the actual Debian votes on systemd (not flamewars on mailing lists), your statement does not seem to reflect reality. Systemd, as well as support for alternatives like elogind, was voted in with strong consensus. https://www.debian.org/vote/2019/vote_002
>Forking by a minority is only practical if the system is made up of small components with standardised interfaces between them.
Systemd is composed of several small components, and its interfaces have become standard across many Linux distros, so this shouldn't be a problem.
>you don't have the Four Freedoms in any practical sense.
It is not reasonable to expect that every single person will have the time and expertise to contribute to any given project, or that every project will be structured in a way that lets them accept maximum contributions. I don't mean to dismiss your frustration with being unable to contribute. I get that. But you continue to have the opportunity to do so and you will for as long as these projects remain alive. The code is not going to magically disappear one day and it's up to you to find the time to actually dive in. Good luck.
Is there any commitment to treating those as stable interfaces, with a decent deprecation cycle around any incompatible changes? If so then that's a big improvement; there certainly didn't used to be.
> I don't see what this has to do with anything. It's open source. If you want to collaborate, you can submit a pull request, work with the maintainers and get them to spend their paid time reviewing and merging your code.
I saw patches submitted and rejected on the grounds that the project leadership had no interest in supporting non-systemd (and/or non-Linux). Even if the leadership does want to accept a patch, that can take months; in my experience it's not worth making a change if you can't run it on your own systems without upstream getting involved. So if a project is large and not cleanly factored into smaller pieces with stable interfaces between them (or at least open to being factored that way in principle), then it's just not worth trying to contribute unless you can work on it full-time.
> It is not reasonable to expect that every single person will have the time and expertise to contribute to any given project, or that every project will be structured in a way that lets them accept maximum contributions. I don't mean to dismiss your frustration with being unable to contribute. I get that. But you continue to have the opportunity to do so and you will for as long as these projects remain alive. The code is not going to magically disappear one day and it's up to you to find the time to actually dive in. Good luck.
I believe in open source not because it gives me a chance to write code but because it lets me fix bugs or change behaviour that's bothering me. It's the "RMS printer driver" situation; if something goes wrong in Linux these days I feel like I'm no better off than I would be on windows. For the moment I'm using FreeBSD (and KDE) and I can still patch stuff without my patches getting broken every other month, but as things integrate more deeply with systemd it seems like a matter of time before the programs I use go Linux-only.
About BSD and non-systemd systems: I actually think it is possible for there to be better compatibility here eventually if someone builds more shims. BSD using Mesa for example is a really good thing for everyone. However FreeBSD and Linux have both refused to budge for quite a long time on various things and cgroups seems to have been the final nail in the coffin for any kind of expectation of modern cross-platform tooling. Everything now is container this, container that. The only non-systemd init that hasn't dragged their feet on this has been OpenRC, and even they still would not make any effort to be systemd-compatible because of objections over other things. So what is the solution here? Do you see how hard it even is to get people to run something like elogind on top of another init with another cgroup implementation? I don't blame the systemd developers for not bothering with that. It is not feasible to expect them to maintain everyone else's init as well as their own.
I checked the systemd contribution graph and it seems they have a large number of smaller contributions from outside developers, so the data does not really support your assertion about it not being worth it. https://github.com/systemd/systemd/graphs/contributors
Also, just going by the data, the repos for KDE (collectively) and FreeBSD (a monorepo, like systemd) are much, much bigger, just as tightly-coupled if not moreso, and get a lot more activity than the repo for systemd. It seems strange that you would pick those projects to contribute to.
You either hammer out a common interface with compromises from both sides, or you implement the parts you want as progressive enhancement from a lowest-common-denominator baseline. It's not easy, it's not glamourous, but it's the right way to do it. Look at how e.g. web standards stuff happens when there's disagreement between different browser makers.
> Also, just going by the data, the repos for KDE (collectively) and FreeBSD (a monorepo, like systemd) are much, much bigger, just as tightly-coupled if not moreso, and get a lot more activity than the repo for systemd. It seems strange that you would pick those projects to contribute to.
I'd disagree with the idea that KDE is more tightly coupled; in my (limited) experience it's very cleanly factored and it's clear where the interface boundaries lie. It's a big project but if you want to fork and patch one part of it you can do that.
FreeBSD is theoretically more tightly coupled than GNU/Linux, but in practice I've found a lot less of the kind of interface churn that you see in Linux. Linux went through two or three rounds of different audio APIs where on FreeBSD you still just use OSS. I've lost count of how many times the way you do network config on Linux changed - I genuinely can't set up a network on Linux any more - whereas in FreeBSD it's all just ifconfig. Linux forced a quick migration to HAL followed by an equally quick deprecation in favour of udev; FreeBSD is still using HAL. There's just so much more commitment to stable interfaces, both in theory - a deprecation cycle for even the kernel ABI - and in practical experience. Or maybe it's just that the project doesn't have enough manpower to churn as much as Linux does, but either way the end result is that I can patch my system and expect my patch to keep working.
What do you mean with project leadership. If someone submits a patch, the maintainer(s) of the git module can obviously reject it if they don't want to maintain it. Unless they're persuaded because it's better for the project as a whole.
However, you talk about "project leadership". That doesn't really exist. There's loads of maintainers. There's a GNOME foundation, which mostly handles admin stuff, sysadmin stuff, etc. There's a release team. They ensure the various maintainers make releases, plus verify stuff actually builds.
If you help out in GNOME you're just a person in a big group of people. There's not really any "leadership", just loads of people who often agree, sometimes disagree, etc.
You do realize even the Linux kernel doesn't make any commitments regarding API stability, right?
That's kind of an explicit choice to allow the maintainers to move forward with the limited resources they have without too much red tape.
There's always *BSD, if you want compatibility back to the Sun days in some cases, (Win32 is another option). Linux is about being as modern as possible without having to worry much about backwards-compatibility and all the cruft that comes with having to maintain that.
It does for userspace APIs. Linus is notorious for chewing people out for breaking them (example: https://lkml.org/lkml/2012/12/23/75).
You say "even" the Linux kernel; actually most projects don't and Linux is a rare exception, and I do think it's a bad idea for both technical and nontechnical reasons. (Compare with GCC's refusal to offer a stable API between frontend and backend, which I'd argue is a big reason why LLVM is overtaking it).
> There's always BSD, if you want compatibility back to the Sun days in some cases, (Win32 is another option).
I actually do run FreeBSD these days; one of the reasons this issue still bothers me is that compatibility with BSD is likely to be collateral damage from the systemd push (it's getting harder and harder to run Gnome on BSD, for example). It was particularly frustrating that systemd basically killed off Debian GNU/kFreeBSD just as it was becoming a first-class architecture; that could have been the best of all worlds.
That's because elogind is architecturally all wrong - instead of sticking to the defined stable public DBus interfaces of systemd, and implementing those independently from scratch, what they did was rip the implementation of systemd-logind out of the systemd package and hoping that the intentionally unstable internal interfaces in libsystemd that it relies on will continue to be provided.
> Gnome worked fine long before systemd and systemd has not noticeably improved it (if anything the opposite); there was no technical consensus that the current hard-dependency was necessary
There was a very, very simple reason for why Gnome depends on systemd-logind: nobody volunteered to maintain the ConsoleKit backend, or ConsoleKit itself for that matter, so it bitrotted and was eventually removed.
For most cases you would get better results by networking n cheap boxes rather than trying to buy a single beefy box to support n users on every dimension including perhaps even power usage if you used low power devices.
Contention for io is a problem. Everyone wanting to say decode video simultaneously is a problem. Browsers high ram usage is a problem.
This sentiment has always felt weird to me, as I have always seen UNIX as an OS that has been designed - from the ground up (in late 1960s) to support multiple concurrent users - whether it is via a serial terminal, and now multi-seat, VNC or an X terminal. By today's standards it's not even such a "beefy box" that can easily do that.
It's what I'm about to be installing after a decade on the Ubuntu desktop. The recent snap fiasco is just the final nail in the coffin. The trifecta of SystemD, Gnome3, and SnapD means that I can just use Windows with WSL if I wanted a distro that is maximally intransparent and pushing things onto me (saying this without satire). Actually, gnome3 is really the worst regression here, and it being so strongly tied in to SystemD makes it an easy target to get rid of.
Other contenders I've been looking into: Void Linux, Slackware, and the BSDs (I used to run FreeBSD in the late 1990s/early 2000s). At one point there was even a variant of Debian (userspace) running on FreeBSD, which however was abandonded due to SystemD and cgroups/namespaces invading too much of Debian.
If it doesn't work out I can recommend Void Linux. I've been using it for years as my daily driver since Debian switched to SystemD, both at work and at home.
It's because systemd offers convenient functionality that those dependent packages want to hook into, because previously they were either maintaining all that themselves and seriosly lacking the manpower to do so or it wasn't being maintained at all and just rotting, (i.e. consolekit)
Don't like it? Write your own/Leave.
The fact is you can't go back to it as an individual, because the system has changed and as an individual you're powerless to change the situation at all, especially against an army of developers paid full-time. The latest news from Debian is sysvinit support is no longer guaranteed.
If anything the power differential has gotten much smaller in recent years with things like github, and last time I checked systemd was accepting pull requests. And even though they won't guarantee it, it sounds like Debian also will continue to accept contributions from those who want to spend time trying to support sysvinit. What exactly is the barrier you're having?
That being said, SysV init is (in fact) terrible. I'd say put the effort into something that can supercede systemd some day. That part of the problem is tractable, though success is quite a long-shot given its entrenchment.
That was never the case, especially so for Debian that you mention. In the Debian Hoo-Hah, the choices were van Smoorenburg init+rc, OpenRC, Upstart and systemd; the latter three being the main contenders, as was acknowledged partway through the affair. In Fedora and Ubuntu, the choice was between Upstart and systemd, Upstart having been what they used for some years before systemd.
* https://web.archive.org/web/20141222234706/http://uselessd.d...
Just as a side note for the interested, there is a project named Devuan that launched to keep alive a Debian sans systemd.
(I've never run it myself; I just happen to know it exists.)
The code is there for someone with enough skill to show how bad systemd is right?
It would be impressive if Lennart managed that by himself. Red Hat wanted it done, Ret Hat also has its hands in various open source projects that suddenly started to sprout hard dependencies to systemd. Projects like Gnome were Red Hat is by far the biggest contributor.
Not much an individual programmer can do compared to a corporation throwing its weight around to break things.
RedHat wanted it done once Lennart convinced them it is the right thing to do and Lennart put in the effort to be at RedHat at the right time to have the ability to convince them.
In fact Lennart is one of the few people willing to put in the effort to touch the fundamental building blocks that otherwise rot, but nobody is willing to touch.
> Projects like Gnome were Red Hat is by far the biggest contributor
Yeah, I guess it's not that bad that in FLOSS, the people willing to ultimately sit down and put in the time and effort to actually write the code, even the non-sexy bits, get to steer the project over Hacker News commenters.
That's not the worst outcome out there if you ask me.
Red Hat had zero interest into a new init system when Lennart started systemd. They had just moved to upstart and Red Hat customers don't really care about init systems. Red Hat customers pay for their systems to be stable and to have someone to call when something goes wrong.
Systemd gives Red Hat no competitive advantage whatsoever.
I'm probably not going to make myself a lot of friends by saying this but I don't really understand the systemd opponents.
The components systemd is slowly replacing or sitting on top, most of the low level userspace of linux, is mostly garbage and poorly documented garbage at that. PAM is aweful. I don't even want to talk about ConsoleKit. Cron has one of the worst configuration format ever and I'm certainly not going to miss fstab. I would find journald a step in the right direction even if the only thing it brought to the table was the ability to configure logging from the service file and not have to fiddle with logrotate.
Networkd gives you a nice, uniform and well documented way to configure both interfaces, rules, custom routing tables and vpn tunnels from declarative files. Who in their right mind can miss the hodgepodge of scripts that was there before ?
I think sections 3.5 and 3.6 (all of them, including subsections outlines current problems pretty clearly. Summarized, systemd is complex to reason about, ambiguous in how it functions, and poorly documented in many cases. All the very similar directives with slight nuances paint a picture of people that didn't really understand the problem space as well as they thought they did (and to be fair, nobody did, and they probably had the best understanding, as woefully inadequate as it was). For a sysadmin, who rarely (but not never!) has to dive into the intricacies of unit files and the myriad directives and their nuanced behavior for creating a server but less rarely is affected by odd behavior introduced by systemd, it's a liability.
What we have now is a chance to learn from those missteps and build something even better. Systemd as a catalyst for rapid chance was wildly successful. As an init system that capitalizes on those changes in a way that users (that is, system/distro packagers) can take advantage of, not so much, since it's so wildly complex. The article posits that BPF will be used to create programs to take over some of these tasks, which may be right. Alternatively, we could start pulling out the components that systemd subsumed, and formalizing them into new standards that others could develop for. Whether that means actually pulling out the component (e.g. logind) to a new repo, or just formalizing API is up for debate.
What you don't like about cron format? It is nice, consistent. Same for fstab, the only issue I have with it that e.g. Debian doesn't handle mounting network filesystems after the network is up (there are a couple of those and it shouldn't be hard to just grep the file for those).
As for journald, as long as they keep my log files in /var/log in pure text format it can stay.
The issue I have with systemd is that it tries to fiddle with my files, last year I noticed that /etc/resolv.conf contains some strange line with local resolver, WTF? I didn't install one for sure so why (and it was breaking my network).
Use Void Linux, help Devuan etc. You're not alone, but you're clearly in small minority.
It's not my experience.
Setuping services has been a pain for 15 years for me. Then with systemd it was suddenly easy.
Detractors keep repeating systemd sucks, but that this point, Debian and Fedora apparently decided it was good enough. And I never heard an end user complain.
I've stopped to be open to convincing.
SysVinit was a horrible, inconsistent mess of bash scripts of per-individual-developer variations in quality and not insignificant variations between distros, or even individual services within the same disto, I wouldn't hold that as the cornerstone of the Unix philosophy, that is if one's intention is to paint it in a positive light.
That's not how I remember it. Fedora was pushed to adoption (thread starter characterises it as an internal "power play"), and Debian was presented a fait accompli (article summarises Russ Allbery with "[not] systemd-vs-the-alternatives, but how-much-of-systemd").
> And I never heard an end user complain.
This is ridiculous! I can also claim the world is always dark when I shut my eyes.
Linux mailing lists are full with complaints. A recurring one was an error message that suggests running a command as remedy, only for that command to also error out. (The remedy was formulated wrong.) By then, systemd already had a version number in the hundreds.
"Your issue is user error, because systemd doesn't provide the logical guarantee you want, and adding a hook is outside the scope of systemd." I've run into several instances of this, the worse one being around unit ordering and network interfaces. I eventually solved it with a hack service that put 'sleep 90' between network.target and network-online.target. So much for that startup parallelism.
Another issue is that a dependency loop will cause it to just randomly drop units. There's no insight into this until you notice it happening and go digging. In the abstract, yes there is nothing else systemd could do apart from refuse to boot. Except the problem has only been dragged into the abstract by using a configuration model that punts on real world concerns.
Both of these are exacerbated by magic processes that half-parse crypttab/fstab/interfaces to create runtime units. I don't know if these are from Debian or the upstream. But if the systemd way is better, why haven't those been fully moved to native systemd?
But somehow distro maintainers can never integrate djbware for political reasons. In the case of RedHat, also NIH syndrome.
We had a chance to replace sysvinit with something better. Instead we managed to replace it with something that's horrible in an entirely different set of ways.
The most devastating thing I can say about it is that it feels "enterprise." It's unnecessarily obtuse, complex, and clunky. It feels like a port from some other operating system that's been shoehorned into Linux, or like something designed to sell consulting services and training courses by wrapping very simple things in needless complexity.
We could have ended up with something more straightforward, clean, and simple. It's a tragedy.
Actually, it's possible for more than two things to be true at once. It could be:
1. It could be true sysv init sucked. (Actually, it wasn't that bad as a modular version of local.rc, but parallel and delayed startup was always just a kludge).
2. It could be true that systemd is a good init system. (For me it's a step change improvement sysv init for parallel and delayed start up. About my only complaint is you don't see failures on "systemctl start xxx"; you have to run journalctl.)
3. It could be that adding another 1M lines of code into the systemd source unrelated to starting stuff was an unsound engineering decision - almost inexplicable coming as it does from a senior engineer. Keeping things in separate source databases with explicit interfaces is a well understood way of reducing the chances of ending up with a ball of mud.
4. It could be that moving sending state changes via an RPC interface (dbus) so you have a million places you have to inspect to understand the state of the system and how it got there rather than writing the desired state of the system to disk with a journal where everyone can inspect it was a bad idea.
I don't think this invalidates the thrust of your point, and I don't necessarily subscribe to the concept of monorepos, but Google is a monorepo, so senior engineers are definitely signing off on this sort of thing. Is it a ball of mud though? Actually does seem true more and more as time goes on.
Probably the biggest difference is that well-disciplined engineers utilizing monorepo do sustain those well-defined interfaces. The example of systemd-logind having no interface guarantees (cited in the linked article) is a sign of trouble.
PAM was and still is quite hard to understand but, again, it had enough power and flexibility with @ based includes to allow individual distributions to lay things out sensibly and allow new packages to follow a coherent standard that emerged rather than being forced.
Concurrent init systems helped drive a lot of this, back when there was a sweet spot of Linux complexity increasing faster than processing power could handle. (Nowadays I feel like boot speed is less important, given how many more resources a modern machine has, though it’s still a concern for low resource platforms.)
I guess I’ve just never seen revolution as a solution to hacks and untidiness. Linting, guidelines and policy, with technically compelling reasons to encourage people to follow them have seen much more success, where I’ve seen them deployed, than centralized authority.
These improvements weren't real. There was a long tail of scripts that didn't follow the standard and never would. Every distribution still had their own customizations on how sysvinit scripts worked (e.g. debian with start-stop-daemon, suse with insserv, etc.)
And rc was different on almost every major distro family, which made life even more difficult for no particularly good reason.
And there's a certain point where all these things you are doing are ultimately to paper over the "hacks and untidiness" of everything. The introduction of systemd had two major effects:
1. It broke all non-LSB init scripts (because LSB is all it supported)
2. It introduced a straightforward declarative syntax for system resources (units for services, timers, mounts, etc.)
I would argue that the above two things have done more to standardize the Linux plumbing layer and help people take fuller advantage of it than literally all the linting, guidelines, and policywork we did for the decade prior.
<conspiracy>
I have heard the opinion ventured that systemd is actually a power play by Red Hat to introduce an API layer between userland and the kernel - one controlled by Red Hat. It's not implausible - as Red Hat is funded by selling consultancy services, they have a financial incentive to convert as much of Linux as possible into Red Hat Linux.
</conspiracy>
However, in this hypothetical scenario, the idea isn't to make it difficult to migrate away from "official" Fedora - it's to make it increasingly difficult to build a distro that is not, fundamentally, Fedora. And if everyone is building distros around Red Hat's quirky middleware, which distro is going to work best? Red Hat's, of course. Who are you going to hire to tell you how to manage the thing? Red Hat, of course.
(The logical extreme of this is the situation with Darwin. OSX's kernel+"middleware" is technically open source, but it's basically impossible to build an OS around it.)
By comparison, I don't believe any companies are selling support offerings for open source Darwin, not even Apple.
"One distro to rule them all".
My understanding (I believe detailed in the article) is that Debian, at least, chose systemd because they were forced to if they wanted a reasonable default GNOME experience for their users. If they hadn't, GNOME would have been missing some features that would have been obvious and jarring.
You can certainly argue that they made the wrong call there, and that they should have pushed back (though arguably by the time Debian got there, that ship had already sailed), but "chose systemd voluntarily" is an incorrect simplification.
This (and much of the article) is incorrect. Debian went through an extensive evaluation process at the time, over the course of 4 months, considering several different init systems, and chose systemd based on this detailed technical evaluation. That evaluation also considered sysvinit, openrc, and upstart. All of that evaluation occurred publicly, and is archived on Debian's mailing lists and bug-tracking system.
I realize that you have been one of the more visible members of the pro-systemd side for years now, so I'd definitely like a more coherent rebuttal to both the historical and technical sections.
Of course, you'd probably consider any kind of response to be undignified, and I understand.
As for the rest, I consider https://news.ycombinator.com/item?id=23062725 a good start. systemd was successful because it provided working code that people (including distribution maintainers) wanted to use, for a variety of reasons. This is the type of problem where no amount of architecture-in-a-vacuum discussions make up for actually doing the work and handling all the myriad cases that come up when people use it in practice. systemd did that; the next init system after systemd will need to do that as well.
As a programmer in embedded, Systemd feels like it solves problems we didn't have, and introduces problems we didn't have. I what point is it more conservatism than "if it ain't broken, don't fix it"? God only knows.
Well, every time the Systemd topic pops on HN or Reddit (and even other places where you wouldn't expect it [1]), I go fetch some popcorn... It seems that different people have different needs and different experiences, hence the flamewars.
See https://lwn.net/Articles/808217/ for the outcome of the GR.
I might be misinterpreting the result, but "Systemd but we support exploring alternatives" means, to me, that there is some unhappiness here. To begin with, I believe the GR would not have happened if there was no issue to be discussed?
Debian did _not_ choose to use Systemd. Debian was forced to choose between adopting Systemd despite going against the very principals of Debian, or loosing Gnome. They were put it a very difficult position.
Ubuntu did _not_ choose to use Systemd. Ubuntu, using Debian upstream, decided not to break from Debian. Shuttleworth, the founder of Ubuntu, has a very sublime message about it at the time.
I'm personally of the opinion that when forced into a corner, do not side with whoever pushed you into that corner. The Debian technical committee did not have that resolve, they took the easy way out. Lots of Debian-derived distros followed suit out of necessity.
When it comes to Upstart, it was almost never used except as a wrapper around initscripts. Effectively you booted from Upstart which greedily synthesized all of its 'events' on startup that in turn triggered initscripts, which brought some degree of parallelism and that was about it. The reverse-dependency 'start/stop on' model was too esoteric, and Upstart's state machine brittle. Hilariously enough, the only system to actually use Upstart to its fullest (still does) is ChromeOS.
Prior to systemd and Upstart, as I discuss in chapter 1, distro developers either simply added various preprocessors for initscripts to introduce dependency headers, or they had these vague hunches of 'waiting for Godot' that we could either replace init entirely with D-Bus, put D-Bus interfaces in initscripts, or some other roundabout way that involved grafting D-Bus. Much of this was owing to the 'irrational exuberance' of Linux developers wanting to hotplug the world after 2003-4, but it always remains an on-and-off effort with the people behind it never certain what they want.
Indeed, besides grunt work of optimizing initscripts, I point out that as late as 2009 distros like Debian were still very concerned about LSB-compliance, and that even when they were willing to adopt a new fancy event-driven init like Upstart, it was conditional on reusing tools like insserv to parse LSB headers, and on incorporating initscripts for service management. Also the entire bizarre diversion that the Debian group had of using a Perl script to generate initscripts.
So far as I know, the daemontools approach was never influential. The available dichotomy was between putting lipstick on a pig, and utopian dreaming.
The paradox remains: how can we trust that the same people who did the Wrong Thing so persistently then suddenly turned out and did the Right Thing like they hit jackpot? If anything, 'jackpot' is the right analogy -- it was a fluke. And the same integration failures that were had with sysvinit have now been shifted into wrestling systemd's job model.
"Unification" was a folly, as the distro is still the place that gets its finger pointed at if upstream can't deliver, and upstream insists it's "just" offering a bunch of "modular tools," which ironically was supposed to be the problem to be addressed! Besides, there is still a whole world of Linux devices out there running Busybox, musl, OpenWrt, etc. that are outside the systemd bubble. Or think of all the container images running Alpine (which is also the basis of postmarketOS). Linux is still the same bazaar it always logically must be, but with a new class distinction added to the mix.
I am also confused why you're complaining about the job model and all its churn and conceptual problems when the whole point is that systemd was offering to handle that for the distros, which is exactly what they did and are continuing to do now. You can claim they are doing a bad job (which is probably true in some sense because of the impossibly large scope of the task) but the previous alternative was that nothing was being done about it at all. Which you did acknowledge, but then you went back to the same criticism as before. Why? It seems obvious to me that the gritty technical aspects and/or perceptions and fears about some kind of false dichotomy are not what could have ever influenced this kind of decision. Nobody else is invested enough in it for that to have happened.
The other reality I've seen is that there is no standard way to actually implement a daemon. Nobody does it in quite the same way and using things like daemon(3) don't help. So you can choose to A. patch all your daemons, or B. you can attempt to simplify the task by writing more tooling. Every single Linux distro I've seen, when given the choice, has chosen option B. So really, I don't see anything here that is surprising or farcical at all.
So far as prior art is concerned, see my old post: https://blog.darknedgy.net/technology/2015/09/05/0/
I would also point out OpenWrt's procd as an example of how to have hotplug-capable service management at a fraction of the complexity: https://openwrt.org/docs/techref/procd
Now, this rethinking did indeed come with HAL, when the idea of a master daemon and ad-hoc service manager reacting to hardware state changes was unanimously rejected by the Linux patriciate as wrong. Perhaps with systemd they will one day reach the same epiphany for system and user services. Either way, I'm not terribly personally invested in this anymore -- I just figured I'd wrap my thoughts up on an issue that occupied a good deal of my time years ago.
EDIT: As to B, the problem of process supervision and having an idiomatic way of daemonizing has been solved for a long time: https://jdebp.eu/FGA/unix-daemon-design-mistakes-to-avoid.ht...
Also, things are constantly being rethought. That is quite literally what the churn encompasses. You can't complain about this while asking for it at the same time. I have seen all these lists of other init systems and while they are interesting, they are completely missing where the actual discussions are happening and what is actually being discussed. The core problem has already moved to a different area and that's why it really doesn't matter if they are doing a bad job. I can see you hinting at this in your article but it's not good to dwell so much on things that happened in the past and are already history. I am very grateful that you actually quoted all your sources though, so thank you for that.
The comparison of init to HAL seems not relevant to me; if anything that has made it obvious that they had the right idea but it was in the wrong place, and that init was the right place to put this type of logic after all. Adding to that it also seems very unlikely that the kernel developers will change their position on the responsibilities of an init any time soon.
And about the idiomatic way to daemonize, nobody follows those guides. Nobody. Seriously. It's a complete clusterfuck and writing more guides doesn't help either. There are an absurd number of programs out there that still use pidfiles. Go and do a search on github for "pidfile" and be horrified.
> The core problem has already moved to a different area and that's why it really doesn't matter if they are doing a bad job.
Yes, I agree.
Still, the losers of history deserve their voice, too.
"Nobody" is a significant overstatement. A lot of (certainly not all) daemons that used to only support double-forking and PID files were fixed in the wake of daemontools. systemd and other init systems benefited from this work when they appeared. Still, PID files are subject to TOCTTOU regardless of what you use, so that's a given.
I don't particularly care about who was a "winner" or a "loser" either. In general you cannot get away from TOCTOU on Unix without taking extreme care and avoiding things that would seem reasonable, like storing filenames in other files or using the "kill" command. In practical use I have seen no inits that make any concerned effort to solve these issues at all. That is a situation in which we all lose.
You're basically taking systemd's architecture as an axiomatic premise, so no surprise your conclusion is obvious. I reject the premise, and the fruits of it I go into detail in chapter 3.
For the record I am mostly a Unix-hater, so your complaints about Unix are things I agree with. systemd, however, is simply more New Jersey design with unwarranted pretensions.
systemd, in a way, is exactly the init system that a community as infantile and disordered as Linux rightfully deserves. May they enjoy it.
I'll leave things at that.
If you looked at the discussions that happened during that time (I was there as a Fedora contributor back then, and wow that was a long time ago!), you'll see that we mostly did the change out of weariness of trying to use Upstart features well and failing. And even with that, we delayed it by one release because it wasn't ready during the Fedora 14 development timeframe (it was supposed to be in Fedora 14, we actually launched it in Fedora 15).
Of all examples you gave, this one is the least convincing reason to introduce this Behemoth on our systems.
Yes please!
Pure functional languages are very well suited to declarative configuration, especially where you need significant flexibility not afforded by purely static "data" formats such as YAML or JSON.
The cultures were very different, I straddled them with my hobby working being CP/M->PC->Amiga, and my professional work being RSX-11M->TENEX->TOPS20->UNIX.
Linux (and Minix) started out very much being a PC version of UNIX and PC-BSD jumped in there too. But Linus ceded userland to the community and as the community's membership has grown the people who wanted Linux to be the perfect UNIX lost out to the people who wanted Linux to be an open source version of Microsoft Windows.
It appeared to me that the folks who liked UNIX semantics fled to MacOS, although the complaints on that front are getting louder as well. So here we are.
You know it's written all lowercase, right?
I've seen this scenario played out, and it always fails. The product never ships, of ships in such a bad shape the users have a terrible user experience.
That's not what happened with systemd. It shipped. And after a few iterations, it actually provided a better user experience.
I don't know about sysadmins, but as a simple end user of Ubuntu, systemd is the best user experience I ever had for what matters to me: setuping my services.
It's quite easy to create your unit files, declare service dependencies, stop, restart, and so on. It's declarative, conf is reasonably straighforward to read an understand, documentation is there, and my system boot is stable on laptop and on my servers.
Maybe for "advanced case X", there is problem. Apparently, a lot of power users say there is. But I also know the FOSS community has some people advocating purity over practicality, don't care much about the people with less skills than them, don't understand user friendliness or will fight for their ego. So I have no way to be sure if the problem is real, or just another free software drama.
But at this point I don't care: for the average Joe like me, it's great, and I'm very happy it has been chosen by Centos and Debian as the default.
Apparently nobody in the entire history of init system have managed to provide something decent for, you know, initializing the system. I never heard a herd of devs claiming that one is the fantastic, and they were all terrible to use when I tried.
Well, I don't care for choice in my init system. I want it to work.
And systemd works fine for me.
We Agree.
> And systemd works fine for me.
But it doesn't for me, and I am left with no choice but to suffer with it.
easiest answer is to say get rid of the inter-dependencies, but that's throwing money and time at a problem affecting 1% of users, at no benefit to the rest. Maybe it's still the right answer, but who's to pick up the slack?
I don't know why, but it bothers me when someone's trying to criticize systemd, without even bothering to get the case right. It smells of not having looked at it any more than to form the run of the mill opinion on systemd, while looking at what came before with healthy dose of romantic nostalgia.
I don't want my init system to be a series of garage-quality bash scripts with slightly different error rate tolerances etc, systemd solves that. I don't want my system to just blindly shut down, even if it's not necessarily safe to do so immediately, systemd solves that.
What systemd critics often miss is that it's also a rather valuable resource manager in general. I know that many would say this should not be part of systemd, but the thing is while that sounds nice, nobody built anything remotely as convenient for Linux as systemd in this area.
It's nice to say what systemd got wrong, but I haven't seen anything actually built that can seriously compete with it.
I now actively avoid distros without systemd, because the admin experience of those distros has generally been much less consistent.
We can all talk about the right way to do things, but unless there's code that's clearly superior to what systemd offers me today, am not interested.
One problem is that a number of reinventions were poorly made.
E.g. the log format. Yes, unstructured logs have a ton of drawbacks. Can we take some ridiculously well-tested, reliable embedded database or serialization format (like sqlite) and use it as log storage? Alas.
Another problem is the "I know better" attitude.
Are user processes allowed to run after logout? Which DNS to use during startup? What logging in should look like? In each such case, existing behavior which was widely considered as not broken was replaced by a different behavior, often even without an escape hatch. The new behavior(s) in such cases should be made possible, but the default should be the old behavior.
No wonder I don't run systemd on my desktops / laptops.
All those examples you gave are possible to disable in systemd, and in the case of DNS and user process after logout, they are not even the default.
: Killing of user process are enabled by default in upstream but disabled by default in every distro that I know. And systemd-resolved, even if it is build by default, is not enabled by default even in upstream.
I also remember that even 2-3 years ago the situation was not as nice yet.
I think what became systemd could have been great software, if not for the, mmm, cavalier attitude of its creators (not necessarily personal; it might be amended by RH corporate deadlines).
Face it: the linux userspace has always sucked. Linus punted on it way-back-when (to be fair, he was a solo dev working on a kernel, he had his hands full) and opted for the gnu tools, but those (other than the compiler and libc and a few other spots like grep) have never been best-in-class. It was always just sort of "good enough" to avoid swimming upstream (no pun intended) and trying to swap them out.
Someone decided to make an attempt at making the situation better in one way, doing things differently. Perhaps it was a mediocre attempt, even. (I'm just speaking for sake of discussion here; I haven't read the systemd code, just making a point.). Still, nobody was forced to use it, and indeed many options for avoiding it exist.
Somehow, this ended in a holy war and corresponding schism. It makes no sense to me. Are linux users upset that someone dared question their decades of suffering under a poorly-designed userspace? Agree or disagree with systemd's decisions, it's hard to fault someone for making and releasing software and offering it as an alternative, even if it sucks.
I'm all for experimentation in this space. The old way of doing things was super janky, and I think systemd is an improvement in a lot of ways. I (thanks to a lot more time stuck in my house) have two recent Gentoo installs, one with systemd and one without. They both work, but one works better.
"This is the way we've always done it" seems in opposition to the hacker ethos, to me. Doing things differently to see if there might be a better way seems like it's always valuable, considering that nobody's deleting /etc/init.d/* from your system if you don't participate.
Traditions are not always inherently valuable. Try new things. Discard the ones that don't work for you.
But that's the trouble. The people who make systemd keep integrating it with everything else they make and vice versa, which makes it harder to try new things because instead of things coming in pieces you might replace, it comes as one big interdependent blob.
I recently installed a Gentoo system with ConsoleKit and KDE Plasma, and it works great, not a trace of systemd anywhere. I also understand the original Gentoo guy forked Gentoo and made a new distribution (Funtoo) where systemd isn't a configurable option even if you want it - it just doesn't exist in that distribution at all, as far as I know. In normal Gentoo you can use OpenRC (default) or systemd.
AFAIK Gentoo also supports Gnome with just logind, too. Haven't tried out that config.
People say that if you don't like systemd then don't use it -- and that's what the people who don't like systemd want to do. But not all of them want to compile everything from source like Gentoo.
So use Devuan, you say, or some other esoteric distribution with six thousand users instead of a million. Which is presumably what they are doing.
But they still want Debian/Ubuntu to use something other than systemd by default, because you need a lot of people to use something in order to get good bug reports and get people interested in fixing those bugs and making improvements to it. Use in popular distributions is the only way for the alternative to get those things, which makes people who don't want to use systemd unhappy when the most popular distributions do.
* Systemd was not exactly great when it had to be considered "the" replaecment for SysV init (which does suck), and happened to be much more than an init system;
* The big distros like CentOS and Debian (and then many smaller) had little choice but to adopt systemd, because Gnome had it as a hard dependency.
It's unfortunate that of all SysV init replacements, the evolution chose systemd. By its spirit, it's very anti-Linux: it's often seen as "monolithic" even though it technically isn't, because it tries to control nearly everything, and it does not mind breaking the old ways when it sees fit (and not when the old ways are broken).
Quite likely, we're stuck with it as a mainstream solution for another decade, or two, much as we're stuck with X. (And in two decades, Linux the kernel may become irrelevant, due to hardware and OS research progress.)
Which userspace are you claiming is better then? Linux used to offer the best userland experience in the world, because you had control over every part of it, and since things were loosely coupled you could mix and match at will (e.g. prefer BSD grep or Solaris tar? No problem, install them).
> Somehow, this ended in a holy war and corresponding schism. It makes no sense to me. Are linux users upset that someone dared question their decades of suffering under a poorly-designed userspace? Agree or disagree with systemd's decisions, it's hard to fault someone for making and releasing software and offering it as an alternative, even if it sucks.
They didn't build a better alternative and let users choose to adopt it organically. They made a wilfully incompatible, tightly coupled thing, and then pressured other projects to hard-depend on it (e.g. Gnome).
This is quite literally the antithesis of everything that Linux and Unix stands for. The whole reason people were able to use the GNU tools with the Linux kernel is because they are loosely coupled and communicate through standardised interfaces. Systemd by contrast doesn't run anywhere but linux, and you need closely matched versions of everything: your linux has to match your udev has to match your systemd has to match your dbus has to match your gnome has to match your...
If systemd had been around when GNU and Linux were starting out, the whole project would have been literally impossible. It's a betrayal of everything those projects stand for.
GNOME wasn't pressured in depending on it. There deciding to depend on logind over ConsoleKit. Initially logind could be used without systemd, that changed after a systemd version due to cgroups v2. There's quite a bit of overlap between the people maintaining ConsoleKit and the people wanting to make use of logind, so there was a push by some GNOME maintainers to use logind.
The arguments against were often quite weak. When pointed out that ConsoleKit was unmaintained for years the answer could often be summarized as "why does it need development".
> It's a betrayal of everything those projects stand for.
Stuff like this makes maintainers of things just ignore you. There might be a valid point in your argumentation, but why bother.
(FWIW I use nano.)
Text editors like vi may have near-zero network effects, but operating systems like emacs and vim have plenty of network effects. If I spend months building an amazing interface to git in Emacs but my colleague uses vim, our organisation is hurt.
That's (I think) why people care: they want to collaborate.
> (FWIW I use nano.)
boggle
Historically Linux users where people that actively choose to use Linux based on a combination of technical & philosophical reasons, thus they are more passionate about it.
Most common Windows users get Windows preinstalled from the store where they bought the computer. It is not really an active choice based on reason, therefore no reason to have a holy war.
The more Linux becomes mainstream, the holy wars will become more & more fringe. Systemd is pushing Linux more into the mainstream.
I don't like the that journald goes backwards in space and updates indices -- I like "append-only" attribute -- but at least it writes data sequentially one after another, which makes it pretty tolerant of data loss.
As for "without escape hatch", I don't remember any functionality like that? In fact, systemd is full of escape hatches which are documented and supported. Just recently, I had to modify "is computer online" check -- it was amazingly easy, all I had to do was to drop-in a fragment file for the right service. I remember doing the same in upstart -- it was a total pain, requiring reading the source code spread over dozens of undocumented files.
What systemd is doing with this log storage system is reinventing sqlite, because systemd has a fairly new and not battle-tested database implementation, it has shortcomings.
I would argue that either they use plain text logs like before or use sqlite but they should not reinvent a new db.
That's great, but you've completely ignored theamk's argument that sqlite is the wrong tool for the job.
You currently can't trust systemd log storage at all, even on my own laptop it manages to have some corruption... Querying is very poor and inefficient, indexing is unreliable... And that's exactly what I expect, building a database is hard.
You can ask SQLite to use dot-file locking instead of posix advisory locking. Or you can ask it to simply ignore locking all together. In both cases, SQLite will work on even a broken NFS system, though with the corresponding concurrency limitations.
Is dot-file locking a compile- or run-time option?
It’s my opinion (as an infrastructure type) that centralised binary logging is an absolute necessity above 20 instances.
But that leaves out every person with a laptop, or those people with a very low volume of little instances.
Structured logging is difficult, that’s why logstash is so flexible about input and conversions. And I think that was the impetus for systemd’s journals; to log structurally.
However. The parent is wrong if he thinks decisions should be made under low disk conditions. Having a rotate rule to keep your sliding window of logs to a certain size is fine. But your servers (and laptops, and desktops) should not be taking arbitrary decisions based on this kind of condition.
> It’s my opinion (as an infrastructure type) that centralised binary logging is an absolute necessity above 20 instances.
> But that leaves out every person with a laptop, or those people with a very low volume of little instances.
Yep, it also leaves out people on embedded hardware, and on devices that are in remote locations without access to the Internet, or in a car, or on metered Internet, etc. It's hard to hold in your head just how broadly Linux is used these days.
It sounds like you're arguing to use an even more sophisticated logging system than the status quota of blindly writing bytes to disk.
Was is executed well? No, not at all. In typical FOSS fashion it just changed from one release to the other, and that's it.
Yes, there were probably notes about it somewhere in a long forgotten changelog deeplinked 4-5 clicks below the surface of Debian's and Ubuntu's homepage.
Should have they introduced a deprecation period? I don't know, maybe, probably. (But who are "they"? systemd maintainers? Debian/Ubuntu?)
All good software is based on this attitude. Wait, I hear you interject. You have examples to disprove this. Well whatever it is you might think isn't the "I know better" attitude is probably just the same "I know better" attitude with one layer of abstraction.
Fast forward 10 years and I still cannot find a reliable way to do this with journald, apart from forcing it to output to syslogd and then shipping the syslog logs.
And you know what? It’s fucking awesome. Structured logging is just baked in. I don’t have to parse fields back out that have been smashed into a string. I get full timestamps, machine IDs, pid, uid, gid, and all the rest. I get a cursor that I can use to durably resume consumption of logs between restarts. Shit, the cursor even lets me ensure that I’ve consumed logs even when they’re being aggregated from multiple machines, without a single change to my code.
Sure there are a few warts here and there. Everything has them. But it took less than a week for me to be convinced that it’s past time for plaintext logs to die.
https://www.reddit.com/r/programming/comments/4ldewx/systemd...
Namely that systemd doesn't allow persistent processes started from the shell by default, preferring to terminate them when the user logs out.
This would include processes like "screen" whose entire raison d'etre is to persist after the user logs out. (Well, it has other uses, but this is the main one IMO.)
The stated workarounds - fiddling with some options like "KillUserProcesses=no" in logind.conf &co. - have so far failed.
I don't know whether this situation is a problem with systemd or the distro, but it seems very much a problem with the culture summarised by the top commenter in the above thread, of (paraphrasing) glibly breaking existing workflows then casually brushing away criticism with arguments often boiling down to: "this is the right way, I don't care about tradition or protecting 'incorrect' usage."
And sane developers don't break multi-decade behaviour in the first place.
Top comment on that link:
"""Salient comment: "Or somebody could go find the actual problem @keszybz saw
here - systemd/systemd#3005 - which is: In particular, for my gnome session,
if I log out, without KillUserProcesses=yes I get some processes which are
obviously mistakes. Even if I log in again, I'm much better starting those
again cleanly. fix that, and stop trying to make systemd break the world
because somebody's gnome session doesn't currently exit cleanly."""
> "Shouldn't this code be part of glibc instead of tmux?"
> -- Nicholas Marriott, 2011
> If you want to change how processes daemonize, why don't you change how
> daemon() is implemented [...] ?
> -- Nicholas Marriott, 2016
> * https://news.ycombinator.com/item?id=11798173
> * https://news.ycombinator.com/item?id=11798328
Do you have decent examples of it failing to work? If so, then that sounds like a bug against screen. The link you provide is absolutely huge and from three years ago.
Please give me a simple "stages to reproduce ..." style report. Please keep it simple and short and I'll fill in the blanks if I can and only trouble you for stuff I'm too daft to work out.
remote login, start screen, ctl-A d (detach), log out, log back in, screen -r (resume) - there is no screen session because it was killed by systemd immediately after log out.
(syslog entry: May 3 09:01:25 $HOSTNAME systemd[1]: session-6.scope: Killing process 3290 (screen) with signal SIGTERM. )
But that's hardly the point, the point is the toxic "you're doing it wrong" mentality that infests the systemd project and its adherents. I've run into this in so many ways since ceding defeat and allowing it onto my systems. This is just the latest in a long line of examples.
I just tried it on a fully up-to-date CentOS 8 box (which uses systemd), and it worked perfectly fine: after logging back in through ssh, `screen -r` restored the screen session as expected.
From what I have read, that screen session would only be killed if I had set KillUserProcesses=yes on /etc/systemd/logind.conf, which is not the default.
I understand Slackware still lacks systemd, so maybe that's a better choice for me :)
This behavior is controlled by the KillUserProcesses= setting in logind.conf, and the previous default of "no" is now changed to "yes".
https://github.com/systemd/systemd/blob/2d4f8cf467b6825c9127...
The upstream default has changed, but CentOS still uses the old default.
It's not switched on in RHEL or Fedora. It's likely to never be so.
I agree this is totally unacceptable, and that the inevitable tone deaf response to such complaints is also unacceptable.
I mean all of this is just a difference of opinion between, "I want a process supervisor to manage all my background jobs" and "I just want to double fork and throw processes at init."
The distro is kde neon.
The point isn't that I can't (or the distro maintainers can't) get it to work, the point is that the aggressive systemd default breaks established practice, and that this is typical systemd behaviour: arrogant and uncaring. IOW user-hostile in the fullest sense of the term. It's not like this is an isolated example.
Poettering:
> Note that in some cases it might be a good thing to kill screen sessions when the user otherwise logs out […]
* https://lists.freedesktop.org/archives/systemd-devel/2011-Ju...
For the lazy, here's the context of that cherry-picked sentence:
> In order not to break screen we currently do not set kill-user=1 or kill-session=1.
> Note that in some cases it might be a good thing to kill screen sessions when the user otherwise logs out (think university, where students not logged in on a workstation should not be able to waste CPU), in other cases its a bad thing however (i.e. in yours). That means it must be configurable whether screen is considered an independent session or not.
This is a feature that sysadmins have been asking for. On any multi-user system you run into this crap where background processes for users who are long-since gone just hang around forever because they do weird things or hang and ignored the signal. We implemented it ourselves with PAM but systemd’s solution is a lot cleaner.
And they’re not breaking any POSIX behavior. Nowhere does it say when the system isn’t allowed to kill a process.
Weirdness is in the eye of the beholder. I regularly run programs whose lifecycles are not in sync with my login session. Why do I need to stick around to see a batch job complete?
I can understand that different folks have different backgrounds which changes expectations... but come on... somebody bringing up tmux, screen, etc. should simply end the conversation. "Oh, that is a common and historical use case that I have not considered, today I learned something."
> Weirdness is in the eye of the beholder
It's a windows acolyte mindset. Many people don't imagine that a login is just another program, rather than the parent process.
And the current behavior is literally what you describe. It’s just the quirk of subreapers being implemented recently that daemonizing a process wasn’t local to your session.
It makes zero sense that a process that double forks is reparented by init instead of your session leader. If how things worked currently was proposed today it would sound crazy.
I have to disagree, I don't think I've ever run something where what I intend is "kill this mid-run if my connection drops".
>This is basically the same behavior as Windows
My recollection from when I used Windows Server (admittedly some time ago) is that the default behavior on connection loss or closing an RDP window is indeed the tmux-like behavior. The user has to click a separate "Log out" button, and even then, has to manually confirm that it's okay to kill anything left running.
It may be a good default if you're starting from scratch, but GNU Screen has came out in 1987. (Poettering was 7 years old then.)
> This is basically the same behavior as Windows […]
Comparing it to Windows' behaviour is not going to win you any positive points with me. :)
> This is a feature that sysadmins have been asking for.
As a sysadmin this breaks my own daily workflow. I have 400 VMs that are treated as pets and I probably screen sessions on at least a third of them. (I have another 500 that are cattle-like.)
> And they’re not breaking any POSIX behavior. Nowhere does it say when the system isn’t allowed to kill a process.
Aka, malicious compliance. See also: nowhere does it say that the US President cannot fire the FBI Director. Also: nowhere does it say that the US Senate has to hold hearings when a US President nominates a judge for the US Supreme Court.
But when using remote connections, ie RDP, the default is that your session is disconnected. You have to actively choose to log out of an RDP session.
When using SSH I typically want the same. If I put my laptop to sleep or a networking issue causes the TCP connection to end, I want my terminal session to be restored when I reconnect via SSH.
Oh my god the number of times we had to bounce systems because the unintended background processes of long gone users overloaded the system.
A mid-level admin in 1995 could have easily made a 100 line script to kill background processes from non-logged in users that were running over three hours (or whatever).
systemd brought nothing to this arena.
In certain cases I don't disagree, but systemd does not implement this feature correctly, so using functionality that's easily reviewable from decades past makes sense. If systemd could properly implement the feature, there would be no need for the scripting.
>It has never been safe to kill random processes using a bash script running in the background, on most Unixes (and Linux) it is 100% impossible to do that without race conditions due to the limitations of procfs.
This is hilariously wrong. Lots of UNIX OSes don't even implement the ps suite of tools by using procfs.
Even then, there were no race conditions in this use case anyway.
>You would need to implement this in C for it to have a chance of being safe at all, and even then, you would need to rely on system-specific functionality because there is no portable way in POSIX to actually do this.
It's already implemented this way on plenty of UNIXen. ps and kill is the POSIX portable way to approach this, so you're wrong about that as well.
Almost everything you said above is incorrect, or misunderstanding basic UNIX.
I would urge you to actually check with your system instead of blindly dismissing me as wrong just because your bash script happened to work without error. In my experience BSD-based Unixes get it right and don't use procfs for ps or pkill. They don't have the problem because they use special syscalls for this. But SysV-based systems have historically used procfs to implement ps. Linux also still does. Try unmounting proc and running ps or kill and see what happens. If you can't do it then your system suffers from the problem, which is that you can't safely send a signal to the process after reading it because there is no guarantee that the actual PID will persist in between calls to read() and kill(). POSIX says nothing about this because it doesn't specify procfs, or how pkill should actually be implemented. This is all fair game as far as compatibility is concerned.
There is also the other more obvious race condition in your bash script which can also be pre-empted in between the calls to ps and kill. This can happen on any Unix and is not some big mystery either. PID reuse has been a known problem for decades and Linux finally got a solution to it a couple years ago with pidfd_send_signal. There is also the matter of cgroups but I am not going to get into that because I doubt you will want to hear about it.
The less systems touches the better.
>I would urge you to actually check with your system instead of blindly dismissing me as wrong just because your bash script happened to work without error.
Who said this was bash? The script from all those years ago worked perfectly without race conditions.
>They don't have the problem because they use special syscalls for this. But SysV-based systems have historically used procfs to implement ps.
Makes no difference, procfs works fine for this.
>Try unmounting proc and running ps or kill and see what happens.
You're effectively never unmounting procfs. Also, if you managed to, systemd would crash!
>If you can't do it then your system suffers from the problem, which is that you can't safely send a signal to the process after reading it because there is no guarantee that the actual PID will persist in between calls to read() and kill().
There's maybe 10 lines of code to ensure that logic. Again, middling sysadmin work.
>POSIX says nothing about this because it doesn't specify procfs, or how pkill should actually be implemented. This is all fair game as far as compatibility is concerned.
POSIX does say something about this; read about signals. You stop the process before killing it, and ensure that the start time is the same for the stopped process before the kill. That completely eliminates the race condition, using basic POSIX signals.
>There is also the other more obvious race condition in your bash script which can also be pre-empted in between the calls to ps and kill.
That race condition is eliminated with the logic above. That eliminates the PID reuse race condition, even if it's very rare.
>reuse has been a known problem for decades and Linux finally got a solution to it a couple years ago with pidfd_send_signal.
That functionality is where it belongs now, so instead of coding a logic every time you have to ensure PIDs, it's now handled for you.
>There is also the matter of cgroups but I am not going to get into that because I doubt you will want to hear about it.
Correct, I don't want to hear it from someone who doesn't understand POSIX and basic/intermediate sysadmin work.
But that’s such a useless definition because then all software “provides nothing” since it’s implementation proves that you could have written it yourself.
logind is far more robust than any of the janky shell scripts I’ve seen over the years to accomplish this.
Even then, yes, the functionality of using ps and kill together has existed for 20+ years. Those tools are already implemented, providing the functionality for decades.
In no universe is logind considered robust, and basic scripts from middling UNIX admins have provided this functionality for decades, untouched. Even the "janky" ones.
It’s a scenario that might make sense here. But then again, having a watchdog script just kill leftover processes has been doing this same job for forever. Including this use case in systemd seems like overreach.
Perhaps. I have worked half my career in academia & research, going back to when Solaris (SPARC) workstations were a thing, and this was never a problem.
1. Systemd did not solve any problems I actually had.
2. Systemd introduced problems that I did not have previously.
3. Systemd did not provide me any net benefit, that is, the few benefits it did provide over Upstart/SysV/etc. (easier service configuration and ordering) did not overshadow the issues it caused. Its introduction into my home and professional computing life has been a net negative.
My first experience with SystemD (Ubuntu 16.04 IIRC) turned into a frustrating experience, mostly around trying to mount a NFS volume in /etc/fstab, defined with 'soft' and 'bg' mount options. SystemD would often fail to mount the volume, and as a result would never fully boot to login prompt. Additionally, on shutdown SystemD would hang again forever trying to unmount that volume. No amount of mount option flag tweaking seemed to make much of a difference, so I simply removed the NFS volume from fstab and manually mounted when I needed it.
Happily running MX Linux today and back to mounting NFS in /etc/fstab.
The first message in the bugtracker already has the context of whats happening.
Help me..
apt install openrc && apt purge systemd
Rebooted and it worked perfectly. Now, I get that Debian doesn't really support OpenRC[0] (or sysvinit), and it could break in horrible ways when bullseye goes stable, or get removed entirely, but... I don't see why we need a fork before that happens? It seems like it's a lot of work to maintain a distro fork, when I feel like that effort could be more productively redirected to stronger maintenance and advocacy of OpenRC and/or sysvinit in Debian itself?
[0] Debian's openrc package hasn't been updated in a little over a year, which is indeed concerning. sysvinit does seem to be more actively maintained, though.
The problem isn't systemd's init. It's probably fine.
The problem is the systemd project is taking over the userspace with mutually cross-linked modules.
Devuan was started by a group of Debian maintainers who disagreed with the direction systemd was taking Debian.
There's something extremely beautiful in launching htop and seeing less than 10 processes (including those of root).
In terms of package management... why do you need it? I have no problem maintaining everything with slackpkg and sbopkg along with slackbuilds.org. Sometimes it takes a while to find all the requirements for an application and add them to a queue, but once it's set up it's just sbopkg, click on update, upgrade, and you're good. It's pretty much rock solid once I get everything installed and I haven't missed package management much at all. My main gripe is the old packages in 14.2 but -current has a lot newer stuff. I don't mind waiting though.
Iron grip, aggression, pushiness are needed to lobby systemd out.
Another important initiative is to not to let Poettering, Sievers and co. continue throwing new systemds onto distributives. A preemptive action is needed.
However, if you login to a graphical desktop, that is not a text terminal, and therefore everything is effectively nohup'd: your browser, your chat program, your media player, all the miscellaneous helper processes they spawn, etc. Most people do want that stuff to be automatically shut down when they log out.
Since the kernel's idea of "a login session" is wrong (it only includes text-terminals), and it can't easily be changed for compatibility reasons, systemd implements its own idea of "a login session" that works the way most people (who aren't experts on POSIX job control) expect.
Systemd's revamped notion of "session" includes graphical ones. Yes, that's different from the legacy sense of a mere text-terminal window. But since they've had to reimplement the whole concept of graphical sessions anyway, they could have made them work like textual ones. If you're not running a session manager as part of your windowed environment (almost everyone is these days) the whole thing might break, but then you can just enable the hacky "lingering" mode to make it work and you're no worse off than before.
This is a CADT attitude. Improving existing interfaces while maintaining compatibility is hard; it's also what makes the difference between a serious software professional and an incompetent vandal.
The fact that the logind developers came to a different solution than you, after spending much more time thinking about it and actually implemented it, doesn't exactly imply logind developers are the ones with an attention deficit, or that logind is bad. ("CADT" apparently means "Cascade of Attention-Deficit Teenagers")
Also, never call out others as "incompetent vandals" if you think of yourself as a "serious software professional". This is the kind of toxic behavior that makes communities non-inclusive and leads to impostor syndrome.
The logind developers went against the established wisdom of experience (that big rewrites are generally a bad idea) with the predictable result: high costs (both in migration and in handling outright bugs) for nebulous benefits, with the result that desktop linux is flakier and (understandably) less popular than ten years ago.
> This is the kind of toxic behavior that makes communities non-inclusive and leads to impostor syndrome.
Is that supposed to be a bad thing? We should be less "inclusive" of people who want to rewrite everything. They should feel like impostors. You can't produce good quality if you're not willing to call out bad quality; Linux succeeded (for a time) because Torvalds had high standards and was willing to maintain them.
I'm making a humble request, please do not bring this attitude in open source projects. Really I mean it. It's not helpful and it only makes people angry. You are also misinterpreting the behavior of Mr. Torvalds and confusing things. The kernel developers have actually been some of the most adamant about rewriting major parts of the kernel and breaking internal APIs over and over again (not syscalls) because it's known that the only way to thoroughly improve on the code is to aggressively iterate on it like this. This is actually a major strength of open source: anyone who wants to try to rewrite something can pick up the code and just do it. If it's bad then you throw it away and forget about. If it's good then you keep it. This is precisely how the "high standard" even gets maintained.
I respect your position but I disagree. Projects shying away from criticism in the name of inclusiveness has gone hand in hand with a drop in quality, not just in terms of unwise rewrites but in terms of plain old bad/buggy code - which should not be surprising.
> You are also misinterpreting the behavior of Mr. Torvalds and confusing things.
My point is that Torvalds - historically - used language on the level of "incompetent vandals" freely where appropriate. In a serious software project people are, and should be, willing to state those kind of views very clearly and directly.
> The kernel developers have actually been some of the most adamant about rewriting major parts of the kernel and breaking internal APIs over and over again (not syscalls)
They have; at the same time they've been adamant about the need to avoid regressions, both in terms of maintaining external interfaces and in not ripping things out before a replacement offers feature parity and there's a reasonable migration plan in place. The bottom line is that they caused nowhere near the level of user-facing breakage that the systemd/gnome folks have, and that speaks to higher standards and better judgement.
I want to exclude incompetent vandals from the software community; they are impostors.
Some people consistently make bad decisions. Some of them can change, but others cannot. I do not want the ones who cannot or do not learn to make good decisions to make decisions which affect me (I acknowledge that I myself may be one of those people!).
Quality matters. Reckless vandalism matters too. Breaking nohup was and is indefensible.
No, it's not.
I think you're looking at this in the wrong way. The concept here is one of sessions and the parent-child relationship between them, as well as the decisions parent processes make when they create and manage (or don't manage) child processes.
If you log in to a text console, you end up with a process (like bash) controlling the terminal. If you run normal programs in the foreground, or even in the background (as long as you don't disconnect them from the controlling terminal), they are all children of that bash process. When you quit bash, all its children get terminated as well.
When you log into a graphical session, the login manager (or whatever) will start your desktop's session (which might just be a plain shell script, maybe just your window manager, or might be a full-blown session manager, or something else entirely). Whatever that is, it starts other applications (say your window manager, a panel or dock, desktop manager, etc.), which then can start other applications (browser, chat, media player). If you log out of your desktop, ultimately what happens is that original session-starter (script, WM, session manager, whatever) quits, and it takes all its children with it.
And regardless, the graphical session still runs in a TTY, just not in text mode!
> Since the kernel's idea of "a login session" is wrong
The kernel has no concept of login sessions at all (it does have the concept of "sessions", but they are unrelated to user login). It just starts something as root (init, as PID 1), and from there userspace takes over and does whatever it wants, including the possibility of starting a getty, which can run login, which can setuid() and launch your shell if you put in the right password. (Or run a display manager that does something analogous with graphical sessions.)
So back to the beginning:
> The trouble is that 'nohup' is not a particular state that specifically marks processes that want to survive logout. Instead, the rule is that processes running inside a particular terminal are killed when that terminal closes, and processes running outside any terminal run as they please.
Those two sentences would seem to be contradictory, no? "nohup" is of course not a particular state, though it is responsible for putting a process into a particular state: that of not having a controlling TTY. Which is (greatly simplified) what determines whether or not a process keeps running once the TTY's controlling process exits.
> When you log into a graphical session, the login manager (or whatever) will start your desktop's session (which might just be a plain shell script, maybe just your window manager, or might be a full-blown session manager, or something else entirely). Whatever that is, it starts other applications (say your window manager, a panel or dock, desktop manager, etc.), which then can start other applications (browser, chat, media player). If you log out of your desktop, ultimately what happens is that original session-starter (script, WM, session manager, whatever) quits, and it takes all its children with it.
This isn't how it works though: killing a parent does not kill its child, which is probably one of the bigger design flaws in the original unix (and probably persists because through another design flaw it's the only way to reparent processes, as the insane double-fork 'daemonisation' routine demonstrates). This is one of the things systemd aims to fix: by keeping track of process relationships with cgroups it can kill all the of the processes spawned in a session.
And I'm not sure about your assertion about graphical sessions: pretty much all processes on my PC have no controlling terminal.
Now what happens if it crashes? Repeatedly? Or if a user intentionally manipulates their session to keep running?
Should any user on a shared university computer be able to spawn processes to run for all eternity?
Should your desktop environment crashing lead to all software continuing to run, for all eternity, leaking memory like there’s no tomorrow?
That's for the university IT to decide, of course. Software should focus on providing general mechanism, not policy.
> Should your desktop environment crashing lead to all software continuing to run, for all eternity, leaking memory like there’s no tomorrow?
AIUI, that's pretty much what might happen if you're forced to enable the lingering option to make nohup work. Systemd does improve session management under *ix-like systems, but the fact that it doesn't manage to interoperate cleanly with the likes of tmux and screen is a pretty blatant papercut.
Interoperating with broken hacky solutions is nothing admirable. At some point, you have to introduce a new API, which handles this properly, and introduce a way to give/take permissions for these things (so e.g. zoom doesn’t just run their dataminer forever through the same mechanism).
Default is usually that either any user can have any software lingering, or that any user can authenticate (as you'd do with e.g. sudo) to set this state. This would work via PolicyKit and elevated permissions, similar to the UA prompt on Windows.
You can also white- or blacklist individual applications and services from this :)
That's what I meant with "proper" API in contrast to old tmux/nohupd.
And as service management is the init system's task, it's clear this is something where you have to interface with the init system (as the init system is actually even supposed to reap any process reparented to it to reduce zombie processes).
systemd uses cgroups (kernel namespaces), and if you want something to remain after you log out, then start it in a separate cgroup.
cgroups are control groups, you know, to control processes. Which was simply missing for decades in Linux.
nohup should be enhanced to support systemd, or systemd should provide a nohup wrapper, and just start a new scope (cgroup) for whatever the user launches with nohup.
...
Now, that said, I have no idea why distros and systemd did what they did without much communication, but ... that's usually the Linux way :/
Why should the user be concerned with how systemd happens to implement its session management. If you start a screen or tmux instance, it's unambiguous that you want that part of the session to persist after logoff and to be reachable upon logging on to the system again. That's what screen and tmux were designed to do.
How should anyone know what the user really intends. When I start a tmux in a graphical session and click log out I want the system to stop my things completely, (almost) like I haven't even logged in. After all, my session never was set up to be an always-on server session thingie. The distro never advertised this, etc.
Of course users who grew up on UNIX/POSIX/nohup and on distros that worked in a specific way (ie. without cleaning up processes after log out) expected this to continue.
Debian made systemd default in Jessie, which was released in 2015, and it took 2 years for this subsection to appear there: https://wiki.debian.org/systemd#Orphaned_processes
It either did not bother that many people or no one really did anything to document/address it.
Distros did not care apparently. systemd maintainers were aware that it breaks all and every double-forking self-backgrounding resident stuff, but they accepted the trade off, and even documented it: https://github.com/systemd/systemd/commit/65eb37f8fcf0c82db0...
The graphical environment could advertise that you have launched something that will persist upon disconnecting the session, and give you an option to terminate it entirely instead. (It's after all easier for graphical environments to offer these sorts of "user friendly" hints than for terminal-based workflows.)
I.e.,
systemd-run --scope --user screenThe underlying thing is that we keep driving ourselves to "forward progress" in the sense of a collaborative hegemony, and only in those terms. Either a business wants to own the platform, or the developer wants to build that platform. To do that they have to achieve buy-in from existing stakeholders, but simultaneously reinvent incompatible things. Thus through repeated application of this approach the world of professional software development has aggregated itself into conformance to standards that barely make sense, are poorly specified, and have limited proof of concept, but tick whatever buzzword boxes are relevant to the immediate climate.
If you want to take a real stand, invest yourself in "dead" technologies. Then you can choose whatever you want, and if other people want to follow you on it it's implicit that they are working on a similar problem, and not trying to play the platforms game(else they would be looking for an angle to "modernize")
To us the ashes of our ancestors are sacred and their resting place is hallowed ground. You wander far from the graves of your ancestors and seemingly without regret.
Also, Linux is not Unix and was never intended to be Unix or follow its conventions.
I want to have fun with the computer, not run madly just to stay in place like the Red Queen. Remember to re-read your own comment 30 years from now and see if you feel the same way.
It’s just another example of systemd making everything on the system worse.
[user]@[host]:~$ systemd-run --scope --user screen
Job for run-r0b2a43d19a564e2c9f36b5c9b934420f.scope failed.
See "systemctl status run-r0b2a43d19a564e2c9f36b5c9b934420f.scope" and "journalctl -xe" for details.
[user]@[host]$ journalctl -xe | cat
May 04 03:02:54 [host] systemd[918]: Failed to start /usr/bin/screen.
-- Subject: Unit UNIT has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit UNIT has failed.
--
-- The result is RESULT.
May 04 03:02:54 [host] polkitd(authority=local)[879]:
Unregistered Authentication Agent for unix-process:1504:595444 (system bus name :1.25, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_GB.UTF-8) (disconnected from bus)
----------------------------
I changed my user and hostname to "[user]" and "[host]" respectively, the rest is verbatim.
It's interesting to note that to even see the error report (such as it is) I'm instructed to invoke a special command rather than just look at logs where I might expect to find it, and that when I do so I am locked into some dysfunctional pager requiring me to pipe through "useless cat" in order to cut and paste the uninformative details on long lines.
While the consistent boorishness of my systemd install continues to amuse, the details of this issue and the fact of its resolvability or otherwise by invoking recondite special systemd functionality are beside the point.
Why should I need to know how to run normal and trusted software in special systemd compatible ways that I've previously had no issues with for decades?
As I opine below, the point is the toxic "you're doing it wrong" mentality that infests the systemd project and its adherents. I've run into this in so many ways since ceding defeat and allowing it onto my systems. This is just the latest in a long line of examples.
Ha, if only that behavior was confined to systemd... I am forced to consider it a defining feature of much of the OSS/Unix community.
Systemd is mild compared to many projects and its adherents in this regard.
> enable-linger [USER...], disable-linger [USER...]
> Enable/disable user lingering for one or more users. If enabled for a specific user, a user manager is spawned for the user at boot and kept around after logouts. This allows users who are not logged in to run long-running services. Takes one or more user names or numeric UIDs as argument. If no argument is specified, enables/disables lingering for the user of the session of the caller. See also KillUserProcesses= setting in logind.conf(5).
Strange, KillUserProcess=no worked fine for me
How about we give them some benefit of the doubt, that they are mostly trying to make things better, and instead ask them "what is the technically correct way, in the systemd world, of implementing this functionality?"
I suspect the answer, for screen, would probably require some degree of messing with the user manager daemon.
In the case of terminating programs left running in an ssh shell by a logged out user, it's a pretty simple answer IMO: just leave it alone - if the admin of the ssh server wants to terminate user processes on logout I think that's where the burden should lie and there's no good reason for systemd to get involved, that I can see.
> One thing I’m certain of is that this shift cannot emerge from dilettantes, outsiders and proverbial basement hackers. One does not unseat a platform without already being part of the patriciate that calls the shots on what gets integrated where across the largest nodes in the ecosystem.
It’s a long, turgid “why wasn’t I consulted?” complaint which really just comes back to the question of how open-source projects work. An init system is harder than it might seem at first and requires buy-in from an unusually large number of parties since it affects the OS, everyone shipping daemons, and the operators. If you’re not going to invest that level of engineering time, I don’t see how it’s reasonable to expect an equal voting share with those who do.
An init system is harder then it seems ? You and Jonnax seem to think that unless you made it you have no right to say anything. Have you ever made an init system ? Researched what exactly goes into it ? Sure, there's more then just starting programs. But not much more. The only really complicated thing about getting a [linux] computer from kernel boot to firefox is setting up devices. And even that [udev,"hotplug"] is not complicated as much as it is.. obscure. I know about these stuff, do you ?
I'l defend a newbies right to say anything. Nobody was born with all the knowledge of the world. I will not have compassion for your kind of ignorance.
SystemD is about as much an init as my foot is me. That is the problem that most of those you so casually dismiss have with it. That and that it wants to become mandatory. Could they have just extended udev instead of integrating it ? An i wrong to say that ? Is my opinion wrong because i don't have "Red Hat senior engineer" next to my name ? Programming is just a hobby for me. I don't even have a degree in it. So i'm ignorant on the subject ?
Funny thing.. most of those on teh internets who are vehement in favor of systemD don't know much about how a computer system boots and what goes into that.
edit: Never mind. I see feelings and fancy words are still more valuable then knowledge or thinking. Maybe one day, when computing is not as young of a science as it is now.
I know enough to know that you left out networking and authentication, network service dependencies, filesystems, volume encryption, etc. – not to mention automatic restarts, conditional behavior based on the network or other devices (stuff needs to happen when I leave the office), and the various edge cases which I’ve seen in three decades running Unix systems. Every other operating system Linux is competing with has a much better model than SysV init (Windows has since the early 1990s!), and the world changed from static servers which rarely changed to most systems needing dynamic behavior – what worked in an old server room isn’t a good fit for a laptop!
Again, I’m not saying that systemd is perfect but that most of the complaints tend not to acknowledge the many problems it solved and a decade of angry forum threads doesn’t seem to have resulted in any concerted effort to build a serious competitor. The Debian community had a deep examination of the alternatives and picked systemd because it made sense, not because Red Hat has some brainwashing program.
I remember this being a strong supporter, among others: https://lists.debian.org/debian-ctte/2014/01/msg00287.html
"GNOME in jessie will need systemd as the init system to work with all its features, just like it needs the network configuration to be handled by NetworkManager."
"We’re not really talking about choosing between two competing ecosystems. Rather, we’re talking about whether or not to swap out a core component of an existing integrated ecosystem with a component that we like better."
Also (to try holding back the tide of historical revisionism) let me highlight that Debian's decision was in no way unanimous and the process was not without its own controversy. In reality, the Tech Committee was divided down the middle when it came to their vote on the default init system, and the tie was only decided by the double vote of Bdale Garbee, the chairman of the committee.
You say i left out stuff. I did. A lot more stuff then you wrote here. It would be a looooooong list of things, all of them simple (including "networking" and encrypted filesystems). I mentioned udev as an example of something slightly more complex, but even it is not.
> .. much better model than SysV init ..
You talk as a user, as in an admin. I talk as someone who was once interested in how all these stuff actually work. What i played with (read about, coded, and researched in general) includes init, udev, network hardware management (how ip and iw actually do stuff, excluding wpa), and more. What have i learned ? That almost all of it is done by the kernel. SystemD is not the only.. program that responds to events and goes over a graph and such. In fact, it is not even that hard to do what it does. Remember how i asked why udev doesn't have a generic event.. protocol ? You can scrap udev if you don't use systemD. And why ? For better integration ? The kernel tells udev when a device appears. So why doesn't udev just send a message of some sort ? It could even have a "mode" where it doesn't do anything unless it receives a message telling it to. So why not ? Oh, i don't think it's malice.
> The Debian community had a deep examination of the alternatives and picked systemd because it made sense, not because Red Hat has some brainwashing program.
The debian debate was a clusterfuck of grand proportions. There was one (1!) post on the whole debate that addressed the technical aspects. In the end the conclusion was that it is easier to just use systemD as stuff depends on it more and more (GNOME, notably, funny). RH is a huge company, they are not altruistic in their goals. I wouldn't say malice, but i wouldn't say innocence either.
And again, nothing of substance was said. Just fancy words and feelings. Brainwashing..
PS Fun fact: Xorg depends on udev to send it events over dbus when you plug something related to it. It still works without it, until you re-plug your mouse.
Arch Linux switched to systemd earlier than Debian. And I have not noticed. Arch community accepts breaking changes. Debian? I believe they value stability.
Systemd should have matured, proved its citizenship and qualities. And it would have been accepted. Instead it usurped. People had no choice but to abandon their platforms. Decade of angry forum thread shows that there is a problem. Just like with python3, it's better to learn. And it is especially painful that there was PulseAudio just before. Same pointless change. Removed it and never looked back. ALSA works for me, still.
(also, what were the alternatives? I don't run GNOME myself, but given what GNOME wants do to, it does need a solid session management. I have seen ConsoleKit, it was worse than systemd.)
"GNOME in jessie will need systemd as the init system to work with all its features, just like it needs the network configuration to be handled by NetworkManager."
Replacement for "no problem" should run without problems.
python2 had no problems - there was a clear transition path:
1. "str" => b"str"
2. wait some time
3. "str" is u"str"
SysV, rc.d had no problems - just wait till systmed is ready.
If SysV was as bad there would be same amount of outcry as with systmed
Was it really necessary to make it painful?
Who? Somehow systemd defenders has no empathy and Lennart Poettering is a prime example. No matter right them or wrong they are perceived as threat.
Just be nice. It may hinder innovation but it helps community.
I’ve been running Unix systems for 25 years. All of the things you mentioned were already solved adequately in the Linux 0.9 days, and the BSD’s still use old fashioned shell based inits, and work fine in my experience.
In fact, the only time I’ve had issues with any of this stuff has been in the last ten years, and only on systemd-based distros.
The genius of init systems is that they factor all this stuff out, and just handle starting and stopping daemons.
really?
Let's say (not entirely hypothetically) that you have a daemon. This daemon has a config file, which is large and complex. This daemon's /etc/init.d file builds command-line dynamically, then passes it to start-stop-daemon as usual.
Puzzle level 1)
You made a typo in init file, so that daemon does not start. The "start" command succeeds, but there is no output anywhere, and process does not start. How do you debug it in the sysvinit world?
Puzzle level 2)
Turns out this daemon used LC_ALL (PATH? HOME?) when parsing config, so it works fine when started or restarted from command line, but fails when started as an part of startup process. How do you debug this?
Puzzle level 3)
One day, you happened to be in the server room, watching server boot (you normally reboot it remotely). You have noticed a red message scroll up the screen. This looks important! Too bad it already scrolled off the screen and out of the scrollback buffer too. How do you see it?
The problem of all the init systems is they could be great, but their development just stopped at some point -- and instead of fixing the real annoyances in a general way, they were just saying "this is all fine". Until systemd came and ate their lunch.
> You made a typo in init file, so that daemon does not start. The "start" command succeeds, but there is no output anywhere, and process does not start. How do you debug it in the sysvinit world?
"oh my process failed; let's add `-x` to the invocation."
> You made a typo in init file, so that daemon does not start. The "start" command succeeds, but there is no output anywhere, and process does not start. How do you debug it in the sysvinit world?
as above.
> One day, you happened to be in the server room, watching server boot (you normally reboot it remotely). You have noticed a red message scroll up the screen. This looks important! Too bad it already scrolled off the screen and out of the scrollback buffer too. How do you see it?
`dmesg` or /var/log/messages (or /var/log/dmesg)
you mean, "add -x to the invocation, get back a few screenful of text, wade through that looking for "export" lines and copy them to your terminal, find "start-stop-daemon" line, copy the part of the line and remove irrelevant parameters, and paste into your terminal as well?
I would not call this "easy" or "solved". Possible, yes. Pleasant, no.
Puzzle level 2) inherited env -- I think you made copy-paste error? You just copied the same paragraph again.
Puzzle level 3)
Surprise! Neither dmesg or /var/log/messages or /var/log/dmesg usually contain startup problems! Sometimes the nice daemon might log stuff there, but it is no way universal or required.
The sysvinit recommended way is to call things like "log_failure_msg" and "log_warning_msg", and those only print to stdout, which is the console. That's it. Gentoo's OpenRC does claim to have boot console logger, but it was broken every time I tried to use it.
Are you running any servers? If so, you might want to connect to the monitor, reboot and see if any service has error messages. Or you can use console scrollback after the server is booted, but you might need to recompile your kernel to increase is size, it is pretty small. Or just install systemd, it has full boot logging fully integrated with primary logging. And even a special command which lists all daemons which failed to start.
I mean, yes? Granted I'm not a developer but printing out exactly what's happening line by line with all variables expanded and all logical conditions printed is awesome, it's something we try to reverse engineer into programs with tools like bpftrace and strace.
Conversely an opaque C application which fails has much less tooling in the way of debugging failures. Especially if your package maintainers have stripped symbols.
> Surprise! Neither dmesg or /var/log/messages or /var/log/dmesg usually contain startup problems! Sometimes the nice daemon might log stuff there, but it is no way universal or required.
Actually, that's the opposite. A "bad" daemon might not print anything to stdout when invoked (which goes to dmesg) but regardless, this whole comment just reeks of someone who hasn't actually dealt with systems before 2014.
> Are you running any servers? If so, you might want to connect to the monitor, reboot and see if any service has error messages. Or you can use console scrollback after the server is booted, but you might need to recompile your kernel to increase is size, it is pretty small. Or just install systemd, it has full boot logging fully integrated with primary logging. And even a special command which lists all daemons which failed to start.
Console scrollback is "nice to have", personally I'm very thankful that it does not use memory keeping things there because there is the odd regression when you configure something (like when you tell IPtables to log deny's: that also gets printed to console.)
re----gardless. I think we're talking passed each other. I had no real issue as a sysadmin with a fully fledged bash, because I can debug it. I get that there's issues with that and most people do not like it- I'm not advocating for it, just saying that what you presume to be "impossible" was somehow solved for decades, and quite competently. It may not be pleasant to you but the alternative is not pleasant for me.
The larger issue I have is that "the alternative" is embedded and entangled itself to everything- there can never be anything better produced.
With old init's, it was a small process with well defined functionality so replacing it was easy. (as evidenced by the many init's among unices.).
Anyway, I'm getting ahead of myself.
If you want to see what systemd could have been, I invite you to look at Solaris' SMF. Please ignore the XML-ness of it, it's "very old" in comparison to systemd. :)
> A "bad" daemon might not print anything to stdout when invoked (which goes to dmesg) but regardless, this whole comment just reeks of someone who hasn't actually dealt with systems before 2014.
The daemons never print to dmesg -- they may print to syslog, but not all do. The only daemon that I know of that prints to dmesg is systemd :)
To be specific, let's do a simple example. Here is an actual sysvinit file, /etc/init.d/hwclock.sh . It contains:
if /sbin/hwclock --rtc=/dev/$HCTOSYS_DEVICE --hctosys $HWCLOCKPARS $BADYEAR; then
# Announce the local time.
verbose_log_action_msg "System Clock set to: `date $UTC`"
else
log_warning_msg "Unable to set System Clock to: `date $UTC`"
fi
So here is a simple question: let's say my hwclock chip/driver is unreliable and it did not work on startup for some reason, so that "hwclock" call is failing. My kernel timezone is not set right, the clock is off, and the system does log_warning_msg to warn you about it. How do I find out about this warning?
To make your job simpler, here is how the how this function is defined on my system:
if log_use_fancy_output; then
YELLOW=`$TPUT setaf 3`
NORMAL=`$TPUT op`
echo " $YELLOW*$NORMAL $@" || true
else
echo " * $@" || true
fi
Now, nothing is impossible! I can hack this script so it writes to a temporary file. I can re-execute it again, and hope it fails again. But this gets old, you know? I can spend all day debugging early boot shell, but I did it for a long time already, so now I just want to type "systemctl show --failed", see all the failed services, fix it quickly, and move on to more interesting stuff.
[0] https://www.mail-archive.com/debian-bugs-dist@lists.debian.o...
if you 'echo' it will go to the console.
Your distro can/does log everything that goes to the boot console.
The function 'log_warning_msg' is provided by the sourced bash file: /lib/lsb/init-funtions ; you can find a copy here: http://www.linuxfromscratch.org/lfs/view/7.0/scripts/apds02....
It's just doing echo.
If you add "echo" it will print. I promise.
And anything printed goes to whatever your distro is using to log tty0 console output. (usually /var/log/dmesg or /var/log/messages)
Both /var/log/dmesg and /var/log/messages log printk output from inside the kernel. They do not log echo output from userspace.
Are you sure you are an admin?
None, anymore, that's a little bit the point.
Devuan does, so does Debian 7 and RHEL6.
I just tested it on all three in a VM.
yes, I'm sure I'm an admin.
It solved nothing but the problems made up to justify its existence. And the ones those faux solutions created downstream. Boot ordering problem? Feh. What a gaslight fest. All caused by chasing 1.2s faster boots. What a colossal waste of everyone's time. This is the same cancer that Shuttleworth tried to foist on everyone with that UI abomination. Servers are desktops are laptops are tablets are phones!
Linux users spoke and set the system up to do exactly what it does--use a dominant position to drive out alternatives. They only complain when they suddenly wind up on the receiving end.
Perhaps if you don't like getting the pointy end of the stick, maybe you shouldn't sharpen it in the first place?
Why can't you have a discussion without childish insults?
systemd-as-init-system is not the problem. systemd-as-kitchen-sink is the problem.
It's the tight coupling that annoys many people.
Does udevd really need to be in the same repo? While there may be some nice things about journald, does it really have to be in the same source package? (And why can't it support remote logging with the industry standard syslog protocol? Now I have to run journald and rsyslog. And why doesn't it have an ACID file format that doesn't self-corrupt at times? Why couldn't they just use SQLite or OpenLDAP's LMDB?)
What does systemd 245's systemd-homed have to do with an init replacement?
* https://www.techrepublic.com/article/linux-home-directory-ma...
I like the overall theoretical concept of systemd but it has ugly implementation details like this.
Heck, the author of this article couldn’t even find a design doc or adequate manual for the init part of the train wreck, and it’s been in development for ten years.
I ask not rhetorically, just out of curiosity. It's not a typical type of suggestion for other packages I've encountered with open source, and it feels far more difficult to discover than typical decisions at compile time.
Define “usual people who compile this package”. In my books those are distro maintainers who indeed should be familiar with options provided by the init system they choose. If you decide to include oddballs who compile everything themselves (Linux from scratch?) then again, yes, they are expected to know a thing or two or dig into options or even code from time to time.
Furthermore, even if you compile it, you don't have to use it. Even if you install it, you don't have to use it. (After all it has 2-3 APIs, one of them is the /etc/resolve.conf symlink, if you instead just put 1.1.1.1 in that file, libc will not go through resolved. And similarly for the other entry points.)
It's possible to accidentally compile, install and have systemd-resolved set up on your computer, but it still seems rare that someone can compile and install systemd, yet not know what they're doing.
In this case it should be assumed we’re talking about someone who’s compiling systemd and familiar with the source. And yes, these are flags that anyone who builds systemd from source would be familiar with and know how to set.
It’s the same as when you run a configure script and pass —-disable-feature.
From HHGTTG:
"But the plans were on display …"
"On display? I eventually had to go down to the cellar to find them."
"That's the display department."
"With a flashlight."
"Ah, well, the lights had probably gone."
"So had the stairs."
"But look, you found the notice, didn't you?"
"Yes," said Arthur, "yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying 'Beware of the Leopard.'"
And that's exactly the problem.
For all the complaining I do about systemd, I'd be much less annoyed with it if all it did was PID 1. It's all the other stuff that gets tightly coupled for no discernibly-good reason that poisons the well for me.
Unfortunately, currently systemd-resolved is the only OS level DNS-Over-TLS solution that works for my use cases: roaming laptops and desktops running Linux. At home it's fine because the core router runs AsusWRT-Merlin which does DNS over TLS for all clients using it as resolver.
I don't have a strong opinion against systemd (initially I disliked it moving from Arch's BSD style rc.d). Trend is that systemd is taking over more and more *ctl, seem like its goal is the entire user-space?. Most distros choose to accept and adopt it, and overall (at some point) it eliminated distro differences and lowered the barrier for new Linux users (desktop) and sysadmins.
Personally I started systemd when it rolled into Arch Linux, after which point, rolling upgrade has never gone broke (workstation use case - Desktop Environment / WM), could be skill levelled up after gaining experience though lol...
BTW: I watched `reinventing home directories` by Lennart Poettering (recording @ All System Go! 2019), interesting one. Personally I like to see reinventing wheels to solve legacy problems (people get used to them, sometimes confused, question but don't know how to solve) others don't even consider problems. These helped open my mind and look at things from a different PoV.
[1]: notes on systemd-homed https://sites.google.com/site/imterry/aboutme/profiles/skill...
10 years ago this would have been considered satire
But how large a proportion of machines do you think have mail delivery rely on .forward or .procmailrc in $HOME these days?
In any case, procmail can be executed directly rather than from .forward in most cases, and can be configured to take the rc file to be a file in /etc/procmail-rcs/ as an alternative to $HOME/.procmailrc (at least the version of procmail in Ubuntu; unlike /etc/procmailrc, the files in /etc/procmail-rcs/ will take on the user id of the user the file is named for, like $HOME/.procmailrc).
User crontabs on my system at least (Ubuntu) are in /var/spool/cron/crontabs/[username], not in $HOME anyway. I can't remember the last system I had with user crontabs in $HOME.
People have had systems where relying on accessing the home directory of a logged out user has been undesirable or impossible for a very long time.
I help run a 500 node HPC cluster that leverages procmail. Do I count?
> User crontabs on my system at least (Ubuntu) are in /var/spool/cron/crontabs/[username], not in $HOME anyway.
The crontab(5) may be there, but where is the script that is actually executed? Probably in $HOME/scripts/ or some such.
> People have had systems where relying on accessing the home directory of a logged out user has been undesirable or impossible for a very long time.
I have daily experiences indicating otherwise. I would hazard to guess that many other syadins in academia & research would agree. Even assuming $HOME is in /home breaks a bunch of stuff:
* https://utcc.utoronto.ca/~cks/space/blog/linux/Ubuntu2004Sna...
You do, but you're a tiny minority, and as noted /etc/procmail-rcs/ provides an alternative to putting them in /home. If that doesn't work for you, all you need is to not configure encrypted home directions and nothing changes.
> The crontab(5) may be there, but where is the script that is actually executed? Probably in $HOME/scripts/ or some such.
So don't configure encrypted home directories, then. Nobody is forcing you to enable it. Or you can provide an alternative location, and let your users protect their data better and still be able to use cron jobs. It does not take any flexibility away from you, but it adds capabilities for those who want them.
> I have daily experiences indicating otherwise. I would hazard to guess that many other syadins in academia & research would agree. Even assuming $HOME is in /home breaks a bunch of stuff:
Your experiences with systems that are fine with accessing $HOME does not mean that there aren't plenty of systems where it has been an issue, hence why plenty of software supports alternatives.
Your point of not assuming $HOME is in /home is irrelevant to this issue. The point is that software have had to deal with a subset of users with setups where a users home directory is not necessarily accessible when the user is not logged in for many years. As such most software has options to avoid relying on a users home directory being present already, which reduces this to a relatively simple choice of whether or not you consider the ability to encrypt home directories to be worth making those changes for.
No one is forcing you to change your broken buggy 30-year old shell scripts, you can always continue using them.
How about usernames that start with a digit? Am I still allowed to use those?
* https://ma.ttias.be/giving-perspective-systemds-usernames-st...
Now assuming I have a user called 1000 with UID 2000, and a user called 2000 with UID 1000.
Now what do you think is gonna happen if multiple POSIX-compliant tools call each other?
This is clearly broken, and the reason why BSDs don’t actually implement POSIX cleanly, instead allowing all their tools to take unambiguous identifiers on the CLI, with the # prefix to mark a UID, and have all their tools call all their children with this format.
The POSIX standard is clearly suboptimal here, and the set of valid usernames, and the set of valid UIDs should never have been allowed to overlap.
udev doesn't have to be there, but when a handful of people maintain critical parts of the Linux ecosystem and they saw ways to make their maintenance work more effective, they went ahead and made their work more efficient.
I mean, criticizing systemd/udev/Debian/Ubuntu/Arch on how they did stuff, and the technical merits of how things are is completely fair, but criticizing "why" is strange. They did because they saw that it will help them, their projects, their users. (And they probably understood very well that it won't please all of their users.)
Sometimes vocal minority is just that.
I've read it more as a quirky history writup with some technical and idiology discussions.
FWIW I still think the technical criticism in the article has a lot of merit and what seems "turgid" to you is a nice link collection to learn about systemd minutiae...
they work differently depending on the project.
so this statement:
> requires buy-in from an unusually large number of parties since it affects the OS, everyone shipping daemons, and the operators.
is only true in some cases.
c.f. rcng, which was developed by a very small few people, and is a generally sane improvement on previous BSD init that i've really never heard any complaints against vs the previous method.
same could have been true here. it was not - and the need for 'consensus' was largely sidestepped by the 'market pressure' of gnome being dependent on systemd features, and the leading core desktop environment that people had already attached themselves to, esp in the enterprise market.
This statement makes no sense. It is well understood and has been studied and critiqued by many, including me. Also, I don't know what "social level" has to do with anything here. It was just created for the commercial aspect of Linux and forced into the relevant distro's by commercial interests... it's just that simple.
They had the people because they had the money. The debate about systemd is only partly technical; it's mostly a culture war about how Red Hat is hostilely taking over the free software ecosystem, and how that invokes memories of embrace/extend/extinguish.
I remember reading some advice about not doing rm -rf / on UEFI systems and looking that up, it looked like some MSI laptops would brick if certain UEFI variables were wiped.
I found an issue in systemd's issue tracker. And from what I could tell the ability to write to these variables was required to perform certain functions like rebooting into the UEFI system.
But it looked like that some forum or another had linked to the issue. And plenty of people were up in arms about it without anything substantial to say.
If we're going on technical credentials, I have a few minor ones. I'm not writing dozens and dozens of lines of C every day, and I'm definitely not writing kernel components- but I've been doing low level technology stuff for 15 years now.
Everything from watching access patterns on drives and recombining data to writing/patching memory allocators and aggressively tuning system internals. I'm currently responsible for automating the deployment of the largest always online game my publisher has ever known. I'm relatively technical and highly regarded for my technical competence.
When it comes to systemd; you could argue that I'm "stuck in my ways"; but I've used systemd since fedora 21, I saw bugs you couldn't imagine, I read the code, I saw the hostility of the developers (maybe it's just defensiveness?) and I was a little worried, but ok, it's just fedora, it'll be better in RedHat when it releases... and while a lot of the lower hanging bugs went away for most use-cases; many remained in the edges. Instead of them being fixed though there's been a propensity to build a software eco-system and "system layer" to the OS.
My argument has never really been that "systemd totally sucks" it's that the conversation surrounding it is very hostile.. If you say that you don't like journald (because it's very easy to corrupt and has access drawbacks, encoding issues, tooling is sub-par etc) then people shout at you: "YOU CAN USE OTHERS", but you can never turn journald off.. it's wasted IOPs, IOPs are not free.
The biggest issues I had with systemd is the fact that (unlike Solaris' SMF, which is great btw) it will hide failure from you, and failure can happen relatively randomly because nothing is deterministic. This is not a holdover from being multi-threaded, there is actually a dependency graph that gets generated. But I've never been able to access that. (Maybe that's changed?)
My argument has always been, instead: "systemd's tight coupling of so many components means that we will never be able to have an alternative or successor to systemd". Because re-implementing single components is next to impossible and the sheer scope of the "system layer" is many times more lines of code than the kernel itself.
Anyway the conversation is steeped in some level of tribalism.. Which I truly hate.
You can't talk about systemd without being "pro" or "against" and then people take extreme stances, like "systemd saved my baby" and "sysvinit was trash" (and on my side: "unix philosophy!", and "systemd ate my utility!")
But it was made by guy hired by Red Hat so instead of maturing and having to take actual user feedback before being used in anything significant it was pushed by a guy that vision is limited to "It works fine on my laptop".
Who then decided to reinvent everything along the way like log storage that still takes over hundred of file opens (and tens of seconds) to get "the last 5 log lines of a command" ( https://github.com/systemd/systemd/issues/2460 ), because why the fuck use say SQLite when you can have fun evening inventing your own shitty binary journal format.
I do find a lot of stuff behind systemd useful but the ever-present half-assness of its solutions is annoying. Half-assed logging here, half-assed dhcp client there, quarter-assed ntp client elsewhere, all while basic functionality is still riddled with edge cases.
Still a net positive over having to fix yet another fucked up init script...
Really, the issue here is that "how it used to work" is mostly unambiguous, but I have no clue what you mean by "the spec", and I suspect you're actually referring to a multitude of specs, several of which were only created in the process of writing the relevant systemd components.
I thought the reasoning was to make logs harder (or impossible) to tamper with.
Something something about trading freedom for security.
the binary logs handled by the systemd journal can be "sealed" at regular time intervals. That seal is a cryptographic operation on the log data such that any tampering prior to the seal can be detected.
This might be news to some people.
It seems a lot of people simply don’t want to understand systemd, and go out of their way to avoid learning anything about it.
Could that be considered a social factor?
I’d like to setup a service, which depends on another service, and which must always be running, and if it goes down, it needs to have all forked processes killed, must be restarted, and it needs to run as a specific user.
With systemd? 5 lines or so of boilerplate, independent of distro.
I don’t know about you, but I call that simple.
Now if you are measuring the complexity of the system's (full) implementation, you can't really compare systemd to to SysV-init. You need to compare a systemd-based system to a SysV-init based system.
And then you need to account for the 100s of inconsistently written shell-scripts, all the code distributed in the mess of third party dependencies (sh, bash, Perl, Awk, Python, su, supervisord, cron, etc) to ensure that the functionality of the systems you are comparing is really somewhat equal, not to mention the added complexity of the glue between all those components, and how it may fail.
And when you do that comparison, I do believe you will end up concluding that 1. full systems are somewhat complex, for both systems, and 2. that systemd-based systems are not fundamentally more complex, and 3. systemd-based systems have the benefit of a single, centralized implementation for all these core functions, so that developers don't have to reimplement them inconsistently, and possibly buggy.
With systemd, for better or worse, either your entire init-chain is broken or it isn't, so when it works, you know it works. With SysV-init you have no such guarantees.
Simplicity is an intrinsic property of a product, describing that it solves a problem without needing many parts. Easiness is a property of the product's user experience, describing that it can be used without much training.
That Systemd can make the mania of Linux knobs usable is, IMHO, a huge achievement.
I'm not necessarily a "remainer" on the subject of systemd, but I think it's not correct to sell it as a huge win on simplicity. Maybe it is simple when everything goes right. When it doesn't, though, systemd is fairly nasty to deal with.
On systemd, it is "systemctl mask NAME". On sysvinit? Well, there maybe a file in /etc/default/ -- but each package has its own option name... and some of them cannot be disabled at all.
I hope you're being deeply ironic.
It's several orders of magnitude more complex. It's more difficult to understand and reason about because its internal state is a black box, and the number of interfaces and file formats to understand is again orders of magnitude more complex.
sysvinit had a socket interface with a single message type. It had a single and very simple configuration file (inittab). Everything else was delegated to higher levels. Simple, flexible, and extensible. But above all, understandable in its entirety, and completely predictable.
As a user I 100% disagree. Before systemd I had to learn a few dozen of different config formats for various things * various distros, now all my systems are configured in a reliable & simple way, for init, network, timers, etc etc
For me systemd has hidden some really horrible things behind it's dependency graph (which is a black box) and socket activation issues (for instance, cockpit "listens" on a port, but it's really systemd's socket activation (PID1) and there was a RCE against it).
The configuration file format is, in my mind, mysterious, with random keys in the unit file which have random meaning, and behaviour that is anything but deterministic.
But I've been using systemd for a long time (since Fedora 21). Maybe we're all coloured by not only when we first encountered it but by our distro and how simple they make things? FWIW I'm using Debian and Arch these days and still struggle to "enjoy" systemd, but it definitely functions better on my arch machine.
I don't understand how this is an horrible thing. Thanks to it you can do stuff as
systemd-analyze dot > /tmp/foo.dot ; xdot /tmp/foo.dot
The configuration file format is, in my mind, mysterious, with random keys in the unit file which have random meaning, and behaviour that is anything but deterministic.
systemd might be more convenient from the point of view of an end user. And unit files might seem superficially simpler than a script.
But systemd is not simpler. It provides significantly more complexity, while also providing a lot of features.
systemd is also not more reliable. The previous system, even with LSB dependencies, was usually run in a linear mode with scripts run in a strict sequence. System boot and service startup was deterministic.
If you read the part of the linked article about the sheer quantity and complexity of the internal state managed by systemd, and how it is not deterministic, or even fully understandable even by its authors due to the combinatorial complexity of all the options and how the same settings work differently in different contexts, I'm afraid that I can't do anything further to convince you about the significant design problems it has which directly impact its reliability. Never had it hang irrecoverably at boot, for no discernible reason?
There is a reason people such as myself are appalled by its design. It doesn't even have a comprehensive specification for such a critical piece of functionality. It's defined solely by a single implementation.
The systemd state is a massive hairball. It's effectively a black box, and understanding why the system exhibits certain behaviours is difficult.
The sysvinit state consists of a current runlevel and the state associated with inittab tasks, which is pretty trivial. It doesn't attempt to maintain a mirror image of the state of the whole system, which is largely a constructed fiction.
Proof? sysvinit's surface is anything you can call. It's internal state is everything on the system.
Systemd's surface is the (well documented) commands in the systemd unit files. Each unit is pretty much independent, the dependencies are specified and well defined. Their internal state can be examined and explored.
Read: A confusing mess of different daemons, scripts, and configuration files, which neither worked together elegantly, were simple to configure, nor were easy to debug. Before systemd I would avoid writing custom daemon configurations at all costs because when I did I would need to learn how this version of this distro did it, fiddle with the byzantine mess of scripts, and inevitibly debug multiple subtle issues, now it's basically trivial to get a new daemon running, and I extremely rarely need to debug issues (And I have had to track down systemd bugs due to obscure use cases like one systemd unit modifying other systemd unit configs at the same time another systemd unit with multiple execstart lines is running).
it is so easy to deprecate someone else's skills and knowledge with massive change like this. would it be that hard to respect some of the users?
I think there's a lot of people who get caught up in the merits of systemd and assume that there's nothing better. Or that the alternative is the old sysvinit which was in dire need of being replaced- but that's not true.
Process supervision is totally feasible in init (just look at runit, Solaris' SMF and MacOS's launchd) even Windows has one.
> As it turns out, there was a little-known Freedesktop-affiliated project called xdg-hostname in 2009 which was a start towards creating such D-Bus “utility daemons” for use in desktop widgets and others, all as a standalone project. Had this been followed through instead of having the utility daemons end up being part of the systemd source tree, a good deal of political acrimony could have been avoided – though at the cost of reducing systemd’s leverage.
That is, if they used xdg-hostname instead, SystemD wouldn't be tightly coupled with Gnome, so users could choose between SystemD and InitV, and all this mess could have been avoided.
It’s as if web developers were told now that there is IE, that is prevalent, we can write web apps that only work there ;-)
Can’t be impossible to distill some basics about service startup configuration and dependencies, can it?
Then make it extensible and cry about differences in extension support like we do about per-browser css options, but at least converge on some basics.
If we're talking about init; there are actually better inits already available but you'll never hear about them (runit, for instance) precisely because you don't just leave systemd.
You leave the systemd ecosystem.
This is the main point of the article: systemd as a "job system" is actually quite bad, but the ecosystem has some value in some places and you have to buy the shit with the pig.
I do now. Regressions are a bad thing.
Not for long, probably. I drifted out of this sphere years ago, and came back specifically for systemd's 10 year anniversary, as I felt obligated to at least do that.
Judging by the tone of the comments, 10 years later is still too soon to discuss systemd dispassionately, but when I come back in 10 more years I'll see if things have changed.
Regardless, I did have at least some expectation that people would actually focus on the original post and not brawl in the comment section over tangents that often aren't even part of it.
Oh, well. And hey, if other people down the road read this and it makes something click for them, I'll be happy with that.
Your average HN user doesn't see or particularly care that it makes things easier for distro maintainers, so it's much easier to push the corporate conspiracy angle than to accept that a lot of the hand-wringing takes huge amounts not-very-fun volunteer labor for granted.
As a (Debian) package maintainer for over a decade, this used to be a non-issue. They were write and forget. The average package was just some simple boilerplate.
Maybe the LSB stuff was awful; I never used it, and for that matter I can't think of a reason I would want to. It's the old FreeDesktop.org issue: it's a probably-interesting solution to a problem I just don't have.
1) run the last version of gnome that works without systemd
2) run a patched later version of gnome from another distro that works without systemd
3) don't use gnome, use a different desktop environment
4) fork/patch gnome and remove the dependency yourself
systemd can't take any of those away from you.
If the people actually working on gnome want to add another dependency to gnome, that's their business, is it not? It doesn't break the existing, pre-systemd versions of gnome.
> Indeed, one ought to observe that the most important consumers of an init system are not sysadmins and ops people as commonly believed, but rather distro maintainers. It is their laziness that determines the balance.
Users choose their distribution, ultimately. I don’t agree with the author’s conclusion there.
The article goes into detail about why there was a desire from some distro maintainers to switch to something else. (clean slate).
The reason I accused you of not reading the article is because it's very clear why systemd was adopted.
1) RH Made it, that doesn't automatically make it good, but there was a lot of posturing from the RH devs to get more things tied in to it under the guise of "all distros will use us eventually, they're already in the process of adopting us!"
2) There was a strong desire to get off of sysvinit, many alternatives had come and gone, though most had the desire to be backwards compatible, none could get any real traction, mostly because it was hacks on hacks. Upstart was promising but people didn't use it properly.
3) GNOME added a hard dependency. Small in isolation but then.
4) UDEV was made systemd-only. UDEV being something that underpins a lot of software. A fork was made (eudev) but that was only the beginning because krdb was next to be systemd only.
5) when it came time for Debian to talk about systemd as the last major upstream distribution (IE: one that sysadmins actually deploy and one that other distros are essentially variants of): it was not "systemd or alternatives" it was "how much systemd do we have to take to be able to support most software."
6) When your distro can't take a major desktop environment, that's a lot of points off of your distro.
--
People choose their distro, but their distro often chooses a distro.
If you're a business building your company, you don't trust Gentoo- you run it on RHEL, Debian or potentially Ubuntu these days.
The usual arguments are it is not "Unixy" and then fall back on original designer intent and philosophical attacks, or worse ad hominems against Red Hat or Linus Pottering.
Many people have compared free software to communism. I always used to dismiss it. But, in a way, they were right. Both are movements with a self-conceited historicist endeavor to eliminate exploiting classes ('it is inevitable that the tendency for the rate of profit to fall to produce immiseration as capitalists can no longer profitably invest, leading to a proletarian seizure of power' versus 'it is inevitable that proprietary walled garden development models stagnate as they're overtaken by decentralized communities pooling their knowledge and labor into the most optimal end' and create a perfectly equitable free association of producers no longer having their surplus value be seized by exploiters.
One degenerates into a repressive bureaucratic collectivist (I'm not a Trotskyist, but it's a good term) mechanism for reproducing surplus through either direct requisition or coercive planning. The other degenerates into a way for large-scale cloud providers to appropriate the free labor of volunteers so as to reinvest it back in their walled gardens.
Stalin represented the Leninist wing against ultra-leftists, anarchists and other factions who were against tight top-down organization in favor of 'spontaneous' organizing, and the GNOME/Freedesktop/Red Hat nexus is against the faction in free software that insists on a loosely coordinated and loosely coupled bazaar without a central vision. But in no sense has systemd abolished this, it has only reduced a few nodes at best.
EDIT: Actually I should point out that 'free software' per se is a deontological argument, and it's 'open source' specifically which specifically uses these historicist arguments to justify itself. But then open source was unabashedly an effort for corporate sanitization of free software, which most people forgot after it mostly crowded out 'free software.'
Of course it would take some time to migrate existing unit files.
It doesn't sound like a fundamental critique. Would be nice to turn this into a constructive proposal to fix systemd.
This is a really entertaining writeup, given the subject matter.
1) Breaking working systems should be dissuaded, especially when it comes to human interaction. There's a reason that Microsoft still has a start menu and "File/Edit" menu's
2) There is a certain level of arrogance to the notion that a developer of core infrastructure for literally millions of users can just change things and expect everyone to fall in line without giving credible reason for people to do that.
3) Adding an additional tool to everyones cognitive load should be done with care and delicacy, as it stood there was one set of tools for _all_ file accesses, and there is now a new one.
PS; on a personal note, I'm glad I do not work with you, I find your attitude in the grandparent comment deplorable.
Of course I don’t discriminate at hiring time against people who disagree with me on such a trivial technical detail. Or hunt for them so I can maintain secret lists of undesirables. That would be unethical, probably illegal, and definitely stupid.