The Tragedy of Systemd [video]
youtube.com
youtube.com
Since systemd, I just do a service file.
So despite all its shortcommings, I now have a standard, well documented, init system that I can use accross all the distros I use, which happens to provide all the features I need.
I'm sure all the power user complaints about systemd are legitimate, but I really don't care about those. I just want an easy way to setup custom deamon on ubuntu and centos.
Provide a better system, that solve those problems, and I'll use it. I really don't care as long as my problem is solved.
Same couldn't be said about supervisor, the solution you already had running (well, maybe strike init)?
And it's not installed by default because it doesn't solve the problem of a service dependancy graph.
I doubt that supervisor is not installed because of not solving something it evidently does not want to solve, but that argument at least looks valid to me. Just in case you are not aware, https://github.com/Supervisor/supervisor/issues/122 lists two solutions (wait_for_it.sh and a supervisor plugin that adds dependency resolution).
The more systems something sits under the more places there are for it to cause a problem because it doesn't work quite how the upper components expect. At a bare minimum, you'll often run into problems with the included package management system (and how it may reload systems for security purposes after patching) if you don't take care.
nano sits on top of everything, and the few things that might rely on it use a very standard and not very extensible interface (the EDITOR environment var). It's not really equivalent at all.
Not sure what your point is (well ok, I am, and I agree), but you just lost
Rewatch the first half of the video and imagine it was a bunch of random tools with proper names (like grep, ntpd, syslogd) etc, that had all been built separately but happen to nicely tie into each other, rather than "systemd-blah"
There isn't and wasn't hardly anyone saying that. It is and was well known that there were better things out there than the hacked up sysV style init systems produced by the various Linux distributions.
This is a common strawman that seems to get rolled out any time any discussion involves systemd.
It's not just a strawman, unfortunately. I wish it were.
Systemd on the other hand seems to be trying to take over syslog, ntpd, dhcpd, named, and related with generally poorer implementations. Feature creep seems constant with systemd, for no particularly good reason.
1: https://lists.freedesktop.org/archives/systemd-devel/2014-Ma...
2: https://www.reddit.com/r/linux/comments/6kdcme/why_does_syst...
And the claim that they are 'inferior solutions' is just absolutely false in most cases. It often replaced horrible old tools that haven't been updated in 30 years. It often provides far better interfaces and tooling then the existing solutions.
This isn't necessarily a problem with the systemd project, it's more on the distros that bundle these service, but when you have the same branding across multiple systems, you have to take the bad with the good.
Had they called it "multidnsd" or "ntpclientd" then people wouldn't think "systemd sucks because when MyDistro 9 came along", they've blame multidnsd and let the systemd init system stand on it's own.
You can always disable them at runtime or during compile and use your own though.
The problem with systemd it is not just an init system. Its network-state-aware, device-state-aware container launcher(nspawn), cron, logind and syslog implementation.
The scope creep of systemd is enormous, nobody asked for that when they wanted reliable daemons.
On the other hand, I've had no issues with logind and gdm. I've been interested in trying S6 but given it's quite easy to substitute out the bad parts of systemd, so I just haven't bothered.
Probably the biggest problem with systemd is that it's hurting BSD. Then again, gnome devs are having enough trouble getting things working on Linux... also binary format journals are so dumb. How have they not changed that?
Nobody would make what Lennart did if we had a choice. Those who liked Systemd would use it, and the rest would use the init of their choice. As simple as that.
I agree that a more modern init system is necessary, but the idea of systemd is meh and the implementation is subpar.
The maintainers of Arch Linux have given an explicit list[1] of the reasons why they decided to introduce systemd:
0) it is hotplug capable
1) we can know the state of the system
2) it is modular
3) it allows dbus/udev to go back to doing the task they are meant to do
4) we can reduce the number of explicit ordering dependencies between daemons
5) we get a lot of security/sandboxing features for free
6) systemd service files can be written and distributed upstream
7) systemd is a cross-distro project
8) logind will finally deliver on what consolekit was supposed to do
9) systemd is fast
There are other things that are meant to be left around, as that's their purpose, such as screen and tmux.
I would say the safe solution would be to kill everything in the user session at multiple levels (GNOME, and user session management demon like logind, etc), and to provide a defined safe way for the very few programs that want to stick around to do so. That seems like what systemd came up with as well.
LWN does a reasonably faithful summary.
LWN article:
https://news.ycombinator.com/item?id=19023232 (1/29/2019)
This talk:
https://news.ycombinator.com/item?id=19157293 (2/13/2019)
Another youtube version of it:
https://news.ycombinator.com/item?id=17778202 (8/16/2018)
The fact that I still have yet to see any such examples in the last half a decade seems pretty telling. Wouldn't and shouldn't there be tons of such examples given the rhetoric?
I worked on a system that (among other things) packaged up a Linux system as a network appliance. The device ran multiple daemons. I have built similar things before in a SysV world, and systemd made the following things easy:
* Starting each daemon only when the system resources it needed (e.g. network up) were ready
* Starting the daemons in the right order relative to each other
* Putting daemon log output in syslog
* Restarting when crashed (for those where it was appropriate)
These all required only a few lines of systemd config, and no special coding in the application. For instance, logging was just printing to stdout/stderr and systemd took care of the rest.
You can also manage much more complex dependency situations (like two services being dependent on each other and requiring a restart of both if one crashes).
Systemd Timers are also much more robust and predictable than piling scripts on top of cron, I use them extensively for managing backups on both servers and desktops with automatic reporting for failures.
* http://uselessd.darknedgy.net/ProSystemdAntiSystemd/ (https://news.ycombinator.com/item?id=8488235)
* https://blog.darknedgy.net/technology/2015/09/05/0/
* http://jdebp.eu./FGA/run-scripts-and-service-units-side-by-s...
... and then calling the latter "System V init", when AT&T System V had actually adopted a new system with separate system and service management several years before Linux was even invented.
>and then calling the latter "System V init",
This is what everyone calls it in the context of Linux init systems. You may not like that particular piece of terminology, but please don't give me a hard time for following the standard usage.
Having written and dealt with both; It's about equal in my time and effort to write either in sh or windows ini/'unit' files.
System V init wasn't an ancient straw man for Debian users. It was the init system that systemd replaced.
Then you are unable to distinguish singulars from plurals.
The alternative approach is to use supervisord - it also have service descriptors and does what systemd does. The last time we tried to use it(rhed6 had no systemd) we found it lacking some options important to us.
Yes, it's telling; it suggests you've had your fingers deliberately in your ears for five years.
Maybe systemd is not objectively better or worse. But I'm at least glad there is a standard among Linux systems so I only have to learn the quirks of one init system.
Unit files are much more declarative and self-contained compared to a bash script. And the bonus is that if you need/still want to use bash, you can do so in systemd.
Old fashioned way, when I restart apache having made a typo:
Output of config test was:
AH00526: Syntax error on line 2 of /etc/apache2/sites-enabled/mysite.conf:
Invalid command 'blah', perhaps misspelled or defined by a module not included in the server configuration
Action 'configtest' failed.
[....] Restarting apache2 (via systemctl): apache2.serviceJob for apache2.service failed because the control process exited with error code.
See "systemctl status apache2.service" and "journalctl -xe" for details.
failed!
Systemd makes the process two steps, which you think is worse, but you've failed to address whether the possible benefits of those two steps (error retention, possibly more or different levels of error reporting) have been taken advantage of, and how that looks.
There's a difference between something being worse and just different. So far all I've seen is different, and possibly worse, but you've left out the information that would be required to determine that.
sudo systemctl restart apache2.service || systemctl status apache2.service
Of course if you run sudo systemctl restart apache2.service You get no visual indication it's worked
sudo /etc/init.d/apache2 restart
[ ok ] Restarting apache2 (via systemctl): apache2.service.
This regressive change in behaviour (taking away pertinent information for no obvious benefit) leads to assumptions being made that things are getting worse for normal people. This view is then backed up by the volume of "defending systemd" statements that seem to stem from "scripting is hard".
People get turned off before they even begin to understand what systemd is trying to do, not because of core reasons, but because of one of the first interactions the average person has with systemd is to make their life harder.
There is a benefit. The benefit is that it's logged, always, and through a mechanism that is the same for every process. It's a trade off. We can argue whether it's a worthwhile trade off, or one that even really needs to exist (is there a reason they don't immediately show an error? I don't know), but to say there is no obvious benefit even after I pointed out this specific benefit in my prior comment is odd.
> People get turned off before they even begin to understand what systemd is trying to do, not because of core reasons, but because of one of the first interactions the average person has with systemd is to make their life harder.
Any change from what existed previously that requires a different way of doing something, whether ultimately better or worse, makes the user's life harder in the short run since they need to learn something new.
That's not to say that's all that's happening here. There appears to be a regression in usability in at least one aspect. That said, I sort of understand why they didn't make it output that info. There have been plenty of times in the past for where sysV has reported a service started, but it actually failed very shortly after start. Generally, figuring out why can be painful, because STDOUT/STDERR have been redirected (or likely sent to /dev/null). Keeping a log of the stdout so you can see what happened on that last service action (and not hope it was logged to a log file, which it sometimes isn't...) is very useful. Outputting that the service started successfully is nice. Outputting that it started successfully when it didn't is misleading and more of a problem. Again, that's a trade-off, where it appears they've decided to favor accuracy over convenience in this case.
Yeah, it seems feasible (and maybe a good default) for systemctl to dump out the same last few lines it would for 'status' if an administrative request fails. I don't know why this has to be a tradeoff; systemctl actions could easily both log and print (some limited subset of) the log on error.
It's tiny annoying things like that which leave a bad first impression. Rather than going "hey this system is great, it logs what it tells me", instead I'm left thinking "this system isn't for people like me, it doesn't think I need to know why it's broken"
First impressions are so important.
I agree that it should just output the log messages, but I don't think there is any serious UI problem here apart from just being different. It's also worth noting that the detailed error message you get from Apache isn't universal and depends on the app and how the init script is written. Plenty of sysv services just give you a cryptic error message on failure, e.g. when I restart postgres on my FreeBSD server with invalid configuration it just says "failed" and I have to then enter another command to see why.
It really seems rather sysadmin hostile and replaces tried and true daemons like ISC's named or unbound with a rather poor systemd replacement.
In a "I never actually had to write any code which had significant interaction with existing solutions (openrc, upstart, sysv, bsd, whatever), but I know they exist, so I'll talk about them like they solve the same problems as systemd" way. Which they do not. Want socket activation, double forking, namespaced children, sane scripts/unit files that can build a good dependency tree? You have one option.
Additionally, people see other tools like systemd-network and say "but systemd is doing everything!" Which it isn't. Almost no distros package those as part of the default.
Even if they did, many of the anti-systemd crowd would rather live with a different, slower system (NetworkManager, for example) which is vastly most complicated than what they need. Ditto for systemd-resolved and local caching dnsmasq.
Systemd is not trying to replace NM or BIND or anything. It's basically saying "it's 2019. If you want to be able to work with a system without knowing the ins and outs of 30 different configuration files, and speed up boot time/auditability of your system significantly, you can use these services"
Some things are bad for the community in general. The dependence on dbus (and kdbus), systemd-logind, and other tools aren't very friendly to BSDs to anti-systemd distros. Frankly, from my POV, this isn't systemd's problem. The anti-distros are free to do what they want, knowing the limitations. The BSD guys have written compatibility layers, which you can do, because they're open standards.
Broadly, I'd say that the anti-systemd crowd holds to an idea of "UNIX" purity, as if HP-UX, Tru64, AIX, Solaris, IRIX, and others were uniform. The reality is that they were more different than they were alike, in terms of administrative tools, filesystem layout (is this /sbin/init.d, /etc/rc.d, /sbin/rc.d, etc?) and even software portability. A large reason that GNU autoconf exists is to detect all of the quirks on all of these platforms.
Additionally, anti-systemd people either never used X11R6/OSS/lilo and other old tools, or don't remember how terrible they were. Any large change in the community (X.org fork, ALSA->PulseAudio, grub) draws ire. From the same people who are trying to use syslinux and efibootmgr instead of just using grub2.
Some people would rather debate philosophy than get actual work done, and that's great for open source, even if it's frustrating. But those people never dug into the guts of upstart to make something work, or had 250 lines of sysvinit boilerplate+supervisord saved off somewhere to speed up a process that you can do in 4 lines in a systemd unit file.
* http://jdebp.eu./FGA/inittab-getty-is-history.html
It was like all at one shot - here's this huge and largely non-transparent init system created by seemingly one person who it felt like was shoving it down everyone's throat. My way or the highway style.
That, on top of what felt/feels like a lack of transparency about how it works (I could look at init scripts and understand them fairly easily and how they worked together - systemd seems almost a mystery to me; I just have to trust it works - so far so good I guess).
Maybe I am wrong on all of this - again, if I am (and there's a good chance I am I feel) - please let me know.
> 34:02 binary logs are not a bad thing as long > 34:06 as you have the tools to pull them apart
Really? Did he just "as long as" the biggest downside of binary log compared to text log, that is way fewer tools work on a binary log due to its special encoding?
With the same logic, flu viruses are not a bad thing as long as you are immune to them.
Personally, the video made me rather angry because all he does is set up straw men and knock them down using the same terrible pro-systemd arguments that get trotted out every time someone has a legit problem with its design.Some of the distros that use systems like runit and openrc are using those sustems because systemd doesn't fit their technicals needs, and likely won't ever because of designs that are made without community input.
Oh, sure, he mentioned the Debian developer resignations, but outright failed to point out the reason why they resigned: systemd was being railroaded into the system as a political move rather than being judged on its technical merits.
No. Ian resigned because he tried to ignore systemd's technical merits and tried to used political moves (GR process) to to kill it. As a result, most of Debian's community told him to top messing around with politics. Ian's resignation letter [1] explicitly says that:
> I should step aside to try to reduce the extent to which conversations about the project's governance are personalised. [...]
> The majority of the project have voted to say that it was wrong of me to bring this GR at this time.
[1] https://lwn.net/Articles/621895/
Please stop saying systemd was being railroaded. I have no idea what was going on in Redhat (I do not follow their mailing lists), but at least in Debian, it got on its own technical merits, and Poettering had no special influence there.
(That is not to say it is without problems; there are plenty of them. Still, the lesser of evils...)
Systemd got in on its merits in Fedora as well. (It had to — no amount of railroading is going to overcome something totally dysfunctional, and Fedora was the first adopter.) As I recall, the transition was fairly smooth.
Fixed that for you...
There are a bunch of better alternatives out there.
You can argue that he does that indirectly: He likes that systemd provides something like a service layer, which the tools I mentioned don't really do. Maybe you mean that. That would work, I just happen to think he is wrong there. That's not the job of init, making it a plus point of an init system is some kind of mental gymnastics I don't agree with (though I liked that talk).
The problem with systemd is that it's replaced systems that met peoples needs with a single system that many people consider to be worse.
Perhaps there are some niche environments that systemd is needed, and that's great -- but why break things for everyone else?
> The problem with [old init] is that it's replaced systems that met people's needs [systemd] with a single system [old init] that many people consider to be worse. > > Perhaps there are some niche environments that [old init] is needed, and that's great -- but why break things for everyone else?
So the argument is kind of meaningless without a tie-breaker of some kind; perhaps popularity vote.
I think you're under the impression that your views of systemd's relative suitability are held by the majority of users, but I believe that is incorrect.
If you sell something and people choose to use it to solve a problem, that's great. If you force change on someone, and not only not deliver any benefit to them, but actually make things worse, expect pushback.
Remember too that most people's systemd impact wasn't just replacing init, it was a whole raft of changes from log formats to dns.
Imagine you had a working systemd setup and someone came along and replaced everything. They claim that there are use cases it's better for, but you don't see yourself with those use cases. You would be averse to that change. You then find out that pretty much every distribution going has got rid of systemd. I suspect you wouldn't be happy.