Systemd and the crashing tweet
tim.siosm.fr
tim.siosm.fr
No language is going to produce quality software in a project implementing too much functionality with too few resources. The Go and Rust "safe systems language" hype is strong right now so the language choice is being attacked. The core cause is much simpler to explain.
Systemd (the project) is complex, broad, and deep in scope. It takes a lot of time and effort to arrive at a quality implementation of something like that.
I fully expect the majority of the systemd code to be rewritten, even if the language doesn't change. What we have today is a usable prototype.
When everyone is satisfied with the architecture, feature set, and configuration schema of systemd, after the dust settles, this is all going to be revisited, if there are any users left to warrant it.
There's much work to do, and I bet many of you reading this thread could contribute to getting systemd to where it needs to be. If you use Linux, and can code, clone the systemd repo and start helping!
I believe the first step towards a rewrite is just making an init system+daemon supervisor that (at least partially) understands systemd unit files or a similarly simple solution.
systemd's huge grapple on Linux systems was 95% caused by just by people not wanting to maintain shell scripts (which is understandable).
I'm not a system administrator and I just appreciate that the user-system interface is easy to work with, for the occasions when there's no dedicated sysop in the team.
Nowadays I think it's priceless when I can just create a unit file for my NodeJS and do journalctl -fu my_app.
I think the criticism goes to the leader of systemd, who has exceptional arrogance ( as far as I have read ). Maybe if there is a competition between systemd and you name it, the init system will feel more natural.
systemd is an exercise in high coupling and low cohesion. Most of the valid complaints about it are sniffing around this issue, even if they're otherwise poorly articulated or salty.
The devs have shown they don't give half a damn about criticism towards that point.
Why would I bother sending them patches to change a design I know they're committed to?
¯\_(ツ)_/¯However, it's arguing a strawman that systemd should have been written in a memory-safe language. I never argued this. I know that Rust did not exist and Go barely existed when systemd started development. The reason I called out the memory unsafety of systemd is to underscore how irresponsible it is for it to be parsing messages from untrusted sources as root in PID 1. An alternative to writing in a memory-safe language would have been to use privilege separation, which was very much known in 2009.
> A possible path forward to solve this issue would be to start rewriting parts of systemd or libraries used by systemd in Rust. For example, take a look at the work in progress for syslog-ng (Syslog-ng and Rust).
Unfortunately, rewriting a component of systemd in Rust would require buy-in from the systemd developers due to systemd's tight coupling. In a loosely-coupled system, one can replace individual components with safer alternatives as they become available. This fosters healthy competition and innovation. I'm optimistic that we'll see some really awesome operating system components written in Rust in the future. My fear is that systemd will slow or prevent adoption of these safer replacements due to its tight coupling and the way that it is spreading to more and more of the operating system.
systemd-notify is "parsing", in your scenario by passing the message in the simple key-value format that systemd expects, which is the only "parsing" that PID1 really does, along the notification socket. For everything else, there's generators.
This wasn't a "parsing" bug in PID1, it was a logic bug.
And further, what would be a 'trusted source'? Something that would've done the same thing, passed a data structure that PID1 has to receive and understand?
Your ranting is nonsensical, considering that it's doing pretty much exactly what you wanted.
The important word in the grandparent comment is "untrusted". HTTP and SMTP are simple key-value formats that most people would not consider parsing in a root process in a language like C.
You make a good point about there being no safer alternatives though given the architecture of systemd. What does that suggest about the architecture of systemd?
So you're going to "trust" i.e. deputize some theoretical daemon that will still have to pass the simple data structures to systemd, which won't actually fix, or fix any future incarnations of, the bug that we're discussing.
>What does that suggest about the architecture of systemd?
It suggests that it is, at worst, the same as the status quo on Solaris and macOS.
In reality, the community is filled with people waving their hands and doomsaying against "complexity", ignoring the fact that their suggested actions don't really solve anything.
They seem to be a cargo cult of modularization, if you catch my drift.
Along with all of the comments in the thread suggesting languages that can't fork, and languages that can easily lose track of daemons even when the daemon doesn't double fork, we can see that not every suggestion that sounds reasonable is actually reasonable, or better.
Additionally, there's probably some lesson about `PR_SET_CHILD_SUBREAPER` and what happens when the subreaper they so desperately want to exist crashes.
If systemd were running as PID2, and crashed, children would simply be inherited by PID1, and now we need to communicate everything to the service manager. You can't set subreaper again, after it crashed.
So really, what are we arguing about here?
PHP is installed virtually anywhere, unbelievably fast (really!), achieves most tasks with a reasonable minimum of boilerplate (okay, okay, within reason) while remaining verbose enough to be easy to learn, and makes it possible for anyone to get started.
But the language itself makes you want to... I don't know, exactly - scream, or something? because of stuff like this gem: https://developers.slashdot.org/comments.pl?sid=204433&cid=1... (highly recommended click)
I finally figured it out: this is exactly the same mindset behind systemd! D:
The OP article says that
> systemd should be written in a memory safe language. The obvious picks are Rust or Go.
My immediate response was "no no no no no NO! You need to replace the developers!"
I totally get the idea behind using a kitchen-sink-safe language for Important Critical Stuff™, I really do. And I know that C is slowly going out of vogue for system-level tools of the same order that systemd is in.
What I don't like is that the developers just blindly bumble along thinking everything's fine, denying the importance of issues like these... and get all confused when they get death threats (https://plus.google.com/+LennartPoetteringTheOneAndOnly/post...).
It would be much nicer if the people with the power could just go "ooh, whoa, thanks. Uhh, I don't feel like working on this, but PRs are welcome, and I'll doublecheck if it's been fixed in two weeks." That acknowledgement is really all people want...
That once traced back to the source came from the Sailfish (aka the Jolla smartphone OS derived from Nokia Maemo) IRC channel. And it was uttered as a off color joke about hiring an assassin via Kickstarter after yet another late night problem solving session because they were trying to update to Sailfish to the latest Systemd version.
It didn't come from "haters" or "trolls", it came from within the systemd community. From people actually trying to put systemd to productive use. But it was used as an example of how horrid the Linux community was behaving towards Poettering because they didn't like systemd.
Oh btw, Poettering gave a description of the big that all the recent hoopla is about over at /r/linux. Except that he got it all wrong, and glossed completely over that he was the person that did the change that caused the bug in the first place. And what was that change? The removal of the very kind of check that would stop the bug from happening in the first place.
In contrast we have a recent issue in the kernel, where Torvalds let a patch in that made the kernel oops. The first thing we see is Torvalds blaming himself for not rejecting the patch even though he had noticed tell tale signs of potential problems.
Basically a repeating pattern with Poettering, and Sievers, his right hand man in all this, is to deflect or try to blame others for their screwups.
And you DEFINITELY need a legit source when you're effectively trying to dismiss death threats as justified because they're done from within the same community.
Not systemd developers. Not you. Not me. No death threats.
Which is not to say it's acceptable to make such threats; more a comment on how times have changed.
Okay, I really, really don't like that. I had no idea, either.
Regarding Lennart's immaturity, I'm reminded of https://en.wikipedia.org/wiki/Donald_Crowhurst, someone who joined an around-the-world sailing race only to flounder then begin sending false position data (this was in 1968). He ultimately gave up on the race and his end was unnecessarily tragic.
Please do not misunderstand; there is no suggestion or hint of malice in my referencing this story. I'm mentioning it because it's one of the best examples I know of someone being unable to deal with losing face.
In Lennart's case, because of the reputation/status quo and hate-list he's built for himself, he sadly may never be able to move to a more mature attitude - if he ever does have a perspective change, it may be impossible for him to get others to take him seriously.
I guess the moral of this story is, if you're going to do things that are social (and open source software hacking is inherently very social), maybe fix these things early or something. Before you get famous and make a name for yourself - for the wrong reasons.
Which of these are unfit or confidential? This is an extremely weak argument. It's basically saying that none of the languages besides Rust and/or Go are unfit for system-level software.
(Also it is of course written Ada)
I assume the author meant "esoteric".
All this post-mortem of "I would have done it this way or that way or written in this new super shiny language (which probably has bugs too)" is nothing but a bunch of nonsense. Rant over!
The standards are different for systemd. Indeed they are much higher, just as they are higher for crypto libraries, medical software, etc.
Absolutely right, which is why software, especially critical components, should be written in a fail-safe way that anticipates errors by programmers. systemd is not written this way: just look at the unsafe-by-default umask and the fact that a failure in message parsing can bring down all of systemd, preventing a clean reboot.
I have issues with apache not being restarted by systemd for obscure DNS resolving reasons.
There are some cases of non-systemd-specific packages which are affected, though in my experience, anything brain-dead enough to rely exclusively on systemd is best avoided (GNOME seems most affected).
There are several alternative init systems.
Debian and Ubuntu can be treated in this way, along with derivatives. Slack and Gentoo as well as I recall. I've given consideration to BSDs, but much prefer GNU userland (and yes, I know that's an option).
I'm hoping Red Hat collapses and/or Linux generally comes to its collective senses.
As systemd has in large part been adopted over a manpower issue, expect the ability to opt out via pinning to vanish soon enough.
I'm profoundly disappointed in Debian's decision, but one consequence may be that the worse elements of systemd's design are in fact abstracted out. There are several shim frameworks which can provide systemd-like capabilities, where required, without the full stack being required. On alternate Debian kernels (BSD and Hurd are both targets), this is required due to systemd's own dependencies on the Linux kernel itself.
To a large extent, systemd has taken the joy out of using Linux, and sunk my trust in both kernel and distro development.
If you stick with a tiling WM and don't install GNOME/KDE/whatever, the compilation is far from being a persistent problem, and I say this as someone on a usually <1 Mbps Indian internet connection. (Except for Firefox, that is. But you can get a binary package for that.) I do worry about unfixable package management trouble in the future, though...
apt-get install sysvinit-core systemd-sysv-
reboot
Package: systemd-sysv
Pin: origin *
Pin-Priority: -1
In 2014, the project voted on whether packages could require a specific init system. My read of the results[1] is that a significant minority of the project thinks users should be able to choose their own init system.
Systemd is the new vim vs. emacs
The road to the GNU/Linux unification, was forged by the great wizard of Torvalds. Torvalds powerful use of the: fuck this code is ugly, fix your fucking mistakes, don't break shit via the kernel ... kept the flame of the kernel burning
Meanwhile, across the lake from Linux Use-Land, in the industrial city of Enterpriseyness, a small section of Linux Use-Land citizens formed their RedHat, to show the world how amazing life across the lake is.
They initially served Linux use-land as respected/renowned citizens, but eventually the cities 'hungry' citizens, known as "shareholders" and "government" consumed the Hat.
Beyond consumption, a brainwash-attack was launched by the dirty robber-baron known as "MicroSoft", in which RedHat was tortured into the philosophy of "embrace, extend, extinguish".
RedHat knew that Torvalds was growing weak, as his "Linux Foundation" depended on Enterpriseyness for money. Together, a task-team from Enterpriseyness set out to explore infiltration of Linux Use-Land. This task-team included: NSA, CIA, GCHQ, MicroSoft, Google, Facebook, Cisco, et. al (from the leaked documents of Enterpriseyness arch-enemy: WikiLeaks, a non-redacted version also showed RedHat to be in this task-team).
Enterpriseyness started their exploration via the Gospel of Hacker News. This Gospel was a powerful driver of the future, as reaching the "front page" of said Gospel would ensure eternal bliss and success (and entry into tech-heaven: Silicon Valley). A popular fable was the "Docker", in which this whale-hunting organization showed that with enough good words from the Gospel, hijacking PID1 was plausible.
As such, the CIA activated their sleeper-cell in the Linux Use-Land, known only as "Poettering" in leaked-documents and the trojan-horse called "systemd" was born. The systemd is now being built to consume Linux Use-Land and create a new colony for Enterpriseyness.
PS. This is fiction + written for humour :)
I was reminded of this video by Paol-Henning Kamp (2014) when I read this. It sort of gets at the same set of ideas.
(Feel free to share similar videos, anybody...?)
Thousands of Kernel contributors are Jedi Masters.
RedHat is the Empire. Systemd is the Deadstar?
:)
# ps aux | wc | awk '{print $1}'
200One of the headaches of the old SysV init system was you had specify what order daemons started and when they started in the boot process.
The login daemon started while the OS was still in single user mode. Then you'd start the logger. Now you bring up multiple users, and the network stack. Maybe NTP, DNS resolve,run as separately users? They definitely need system log.
Systemd you just state what your daemon already needs running and it'll on boot resolve the daemon start order.
If you've only ever lived in a package managed eco system and never rolled several of your own daemons you likely never encountered this issue.
The old Apache initscript (for just one egregious example) was a huge ball of shell script wizardry that checked all sort of things on startup, handled restarting/reloading based on the command and some other factors (graceful restart is only possible under some circumstances, etc.), kept up with PIDs, and cleanup. The systemd unit file for Apache is 17 lines long, minus comments; it is declarative, looks like every other unit file, and handles all the same stuff, and better than the old initscript. The complexity is gonna be there. These systems are complex and do complex things. it's just a question of what layer the complexity shows up in. systemd pushes it all down into systemd where most users and administrators and packagers and developers never have to see (much of) it.
And, as you note, run levels and process order are another layer of complexity pushed onto the administrators that could be pushed into the init, but never were under SysV initd. Again, it's handled by humans who have to have a lot of knowledge about the system, the service in question and its dependencies, in order to write the initscript and put it in the right place in the boot order...and there's room for mistakes. I've been packaging system level services long enough to have had many experiences getting it wrong in subtle ways.
Maybe I'm just getting old and grumpy. But I have a Linux 2.0 kernel machine that has been running a LAMP stack for over 10 years that boots in one tenth the time my Ubu 16.04 machines - that run the same LAMP stack - do.
Ah for the old and simple days...
Do try out FreeBSD then -- an OS for the old and grumpy. Systemd free. Uses an init system written in sh(1) that just works.
And yes, sysadmin will still need to tweak things, because a distribution maintainers cannot predict all the possible dependencies between services. One could never forsee that Mailman would fail to run when OpenLDAP was stopped, and this is what I had in one place.
Coming from a group of people suggesting languages to write PID1 in that are without the practical ability, and necessary in this context, to fork().
Or the suggestion to, at its core, make systemd a subreaper even though most certainly don't know that they're suggesting it, or what it entails.