Rethinking PID 1 (2010) | Hacker News Reader

Rethinking PID 1 (2010)

0pointer.de

57 pointsingve74 comments

I remember reading this when it was first posted. The most interesting section, by far, is "Parallelizing Socket Services". That makes it clear that socket activation isn't really about delaying daemon startup or acting like inetd (though inetd did provide some inspiration), but rather about being able to launch services and their dependencies in parallel by making sockets available before full daemon startup. Without that, any service depending on a socket would need to wait for the complete startup of the service providing that socket.

That's one advantage, but socket activation is such a good idea that it's hard to decide what it is "really about": the other main advantages are that your daemons no longer need to contain custom code to open the socket themselves, and as a consequence don't need to start as root if they bind a low port so you can get rid of the privilege-dropping code too, and also if your daemon crashes the socket is still there buffering connection attempts until the daemon restarts. It's really exploiting the fundamental tools that UNIX gives you to the fullest to make things more secure and more robust as well as faster.

It is interesting that after 6 years from the article publication its key concept of persistent sockets and services starting on demand is not there yet.

For example, on the latest Fedora sshd/nginx/php-fpm still bind their listening sockets themselves rather than receiving it from a process manager. Similarly, if one wants to implement a socket activation in Go, it is not a one-liner that just works, but it requires a third-party library to read the socket passed over stdin.

As a result if for performance/security reasons one needs to customize few service socket parameters, then at best one need to look at application-specific configuration language for socket customization and hope that the developers have not forgotten to expose the socket option that is needed. At worst one uses LD_PRELOAD and similar atrocities to customize the sockets.

I think we all got distracted by docker in 2013. nspawn containers can be socket activated, but doing the same in docker is still tricky.

Also, we're all collectively still thinking of containers in terms of tiny-VMs instead of highly customized, tcp processing daemons, aka. websites.

The trickery with docker is that it does not support passing sockets over its HTTP based client-server protocol. But one can still receive a socket from a named pipe.

You can't pass socket fds over TCP, nor should you want to. That particular unixism only works for AF_UNIX sockets.

I would not consult an external library in Go to implement this simple function: https://github.com/Mic92/whois42d/blob/master/server.go#L93

sshd and apache2 for example both support systemd socket activation -- whether distributions actually do that is up to the distros. CoreOS for example runs sshd via socket activation.

On CoreOS the integration is buggy. One cannot start a long running process from sshd as when one logs out, all processes started from sshd are killed.

One can workaround that using systemd-run utility to start long-running operation, but that is a privileged operation. Besides, it violates one of the premises of socket activation that it should be completely transparent.

Didn't CoreOS recently dump systemd?

Now I know there's tons of drama around systemd, and I'm really not trying to re-start any of those flamewars. (please!)

But I never followed the topic that closely, so I'm honestly curious. Looking back on this, he makes a pretty compelling case for his ideas. Were the objections to systemd mainly disagreements with the concepts (parallelizing socket services, etc)? Or was it more the way the ideas were implemented in systemd that traditional sysadmins didn't like? (binary "registry-like" config files, etc)

I only ask because, if most objections were about the implementation, then it should be possible to create a system that achieves the benefits he's extolling here, while keeping a lot of the "Unixy feel" that his detractors prefer. If it's a more fundamental disagreement, then it's harder to see a path that could make both sides happy.

The objections I tend to hear aren't so much about the parts of systemd listed in here, but about the way it seems to have grown to take over the rest of the system.

For example, it has its own syslog replacement, login manager, dbus implementation, network connection manager, dhcp server, udev implementation, boot manager, etc. There's talk of making it a dependency of Gnome, which would make Gnome and its components impossible to run on systems without systemd.

> which would make Gnome and its components impossible to run on systems without systemd.

This is my main complaint. Most programmers know that coupling separate components is a bad idea[0]. After a while, the two components merge; they will begin to interface to lower and lower levels of each other, with the result being a giant mass of interdependent spaghetti code.

Your choice of desktop manager should not, in any way, dictate which init system you use. And with system taking udev and everything else with it, it may be almost impossible to separate out systemd ever again.

[0] I was going to link to the Stevens paper, but even wikipedia has an article about the problems of coupling: https://en.wikipedia.org/wiki/Coupling_(computer_programming...

Nifty.

Seems to me that while unix (and thus Linux) have traditionally leaned towards data or stamp coupling, while systemd is over at common coupling.

Add to that list own DNS resolver and a time manager.

Those are all optional components, though. Why complain of their mere existence? If people want to use them, that's their business.

They are optional from the pov of pid1 yes, but may well not be from the pov of something user facing. And thus you have systemd taking over pid1 because something user facing depends on something that depends on systemd-pid1.

Consider this, upower now is pretty much a wrapper of logind.

The article is great. It's a long tradition for systemd developers:

- They say "We are creating X!"

- People say "Hey, X would be great!"

- They deliver Y, that technically is X, but only technically.

- People say "WTF! Y is very bad."

- They argue "That's why we did X, not Y."

I'm sure systemd developers believe it is the best way to do what is described on the article. But nobody asked for a completely tangled Linux startup.

> binary "registry-like" config files

Just FYI, systemd doesn't use and has never used binary config files. No idea where that myth came from.

Oh, well that's good! Maybe people are talking about his idea for replacing init (shell) scripts with C programs?

It uses binary logs, which is a common complaint.

Its configuration is actually less bad than modern Gnome etc., but is still a great increase in complexity.

A better init system was needed. A better init system is still needed. The trojan horse of systemd definitely wasn't needed. Hopefully it will spin off into its own OS at some point to compete with Ubuntu(sic)'s userland on top of Windows and leave GNU/Linux to develop something that is less of an annoyance.

How is replacing buggy shell scripts that all duplicate the same functionnality by a simple ini file per service "a great increase in configuration complexity" ?

> How is replacing buggy shell scripts...

Barrels of ink have been spilled on this point. You need to read the discussion surrounding this topic to understand that only the most ancient or poorly maintained RC systems were piles of redundant, buggy shell scripts. :)

In this case "the most poorly maintained" happens to be the most used. Only on BSD have I seen rc.d scripts that are not full of redundant boilerplate. Although of course I've not checked every distribution there is, but on those i've used and still use I find a lot of scripts that reinvent racy ways to deal with pidfiles and to let the environment of the shell from which you call the script leak into the service.

I agree that it doesn't mean that rc scripts can't be written properly, but it is a rare occurence on many distributions.

Some issues that I've had with SysV init scripts that disappear with systemd :

* a service having different fd limits when started manually than when started at boot because ulimit values leak from the shell from which you call the init script to mysqld

* a service suddenly having encoding problems because the person who restarted it used a different locale than the one who initially started it and the environment leaks

* countless scripts that fail to properly kill the process

* restart methods that randomly work because it tries to start the new process while the old one hasn't been killed

And all of those on popular distributions. All of those problems go away with systemd.

Not saying it's the ultimate solution to everything, it has it's faults and I'm open to considering other systems but SysV init had to go.

> Only on BSD have I seen rc.d scripts that are not full of redundant boilerplate. ... SysV init had to go.

As I mentioned, this has been discussed widely elsewhere and I'm disinterested in engaging in rehash #1,020,102 of the same tired topic. But, to summarize:

* If your service requires non-trivial work done as part of startup, reconfigure, or shutdown, you must use a script or other external program to get that work done as systemd's unit files will not provide the features required to help you.

* Services that require complicated actions as part of startup, reconfigure, or shutdown will necessarily have complicated code to handle those actions. There is no way around this.

* Only on systems that used both SysV init and SysV RC did you find that gobs and gobs of copypasta. Every distro I've seen over the past decade+ has factored out common functionality into libraries and kept copypasta in the distro-supplied init scripts to a bare minimum. [0]

> ...I'm open to considering other systems...

Check out OpenRC. The minimal service management script isn't much larger than the equivalent systemd unit file and is no more difficult for a complete newcomer to read.

[0] Can you find egregious examples of copypasta and/or scripts that handle edge cases poorly? Sure. But you can find that stuff in any sufficiently large sample of code. Additionally, pushing the required startup scripting out from the distro maintainers to the maintainers of each individual project seems like it greatly increases the chance that a given wheel will be reinvented.

The MySQL open file descriptor limits tale is one that I hadn't heard before. If you know of a full write-up, I might add it to the list.

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwa...

whenever you read a pro-systemd post, you'll see things like "buggy shell scripts" or "just change the logger!" pop up a lot. keep track.

I'm a packager. Before systemd the init scripts were different across distributions. Now they're often upstream and you just package it.

The buggy script problems: trusting the .pid file too much amongst other things.

I'll mentally add a ";)" to your closing sentence. :)

Yeah, I'm quite familiar with the poorly-informed pro-systemd and anti-systemd arguments. I'm also fairly familiar with a variety of init and RC systems and have -thanks to vezzy-fnord and others- learned waaay too much about the history of <star>nix init/rc systems. (There's a lot of wacky stuff out there!)

Cheers, man!

So, one of the other things I hear people have issues with is the bloat of systemd, about how many other systems and services it has consumed and brought in under it's umbrella.

With that in mind, what would a better init system look like for you? systemd-lite, with lots of the questionable feature creep removed and really brought back to it's PID1 beginnings? Something back to sysvinit with it's configuration-as-code, but with modern languages and a general freshen up? Or something else entirely?

It's previously been very hard to get meaningful discussion about this (on both sides of the divide!) because each side is so highly polarised.

> With that in mind, what would a better init system look like for you?

I'm not the OP, but OpenRC is nice. I'm currently suffering from a round tuit shortage, so I haven't waded through the Debian Jessie discussion to find out why it wasn't a contender for their default init/rc system.

> I'm not the OP, but OpenRC is nice. I'm currently suffering from a round tuit shortage, so I haven't waded through the Debian Jessie discussion to find out why it wasn't a contender for their default init/rc system.

It was a contender; it appeared on the Technical Committee ballot alongside sysvinit, upstart, and systemd. Seven of the eight members of the technical committee indicated a preference for OpenRC over sysvinit, just not over upstart or systemd.

See https://bugs.debian.org/cgi-bin/bugreport.cgi?msg=6729;bug=7... for the results of the vote.

Ah! Thanks for that link and the clarification.

As far as I remember, the drama didn't start in 2010. I'm not sure if it started with the epic Debian tech-ctte discussion in late 2013 or earlier than that, but probably not much earlier. The blog post that first introduced it might not be the best place to look for reasons for all that drama.

The drama started very early. I've been seeing the same arguments over and over again since ArchLinux and Fedora adopted it, and the adoption by Debian just got people to repeat them.

The drama started around the time of Poettering's hijacking of someone else's 27c3 speech, accusing people that don't use logind of "hating handicapped people"[0], and continued from there.

0. http://www.youtube.com/watch?v=ZTdUmlGxVo0&t=23m00s

It's the log files that I think got to them. Also, Lennart got a lot of flack for PulseAudio, which I think is a great pity because I really like it. Feel free to penalize me for this, if it makes you feel good go at it :-)

What (if anything) are people's thoughts on runit? I started trying out Void Linux a couple of days ago, which uses it by default. I haven't spent a whole lot of time reading up on it, but from my initial understanding of it, it seems to avoid the issues that people seem to have with both systemd and sysinitv. (Of course, it might just be that it's not used by enough people for anyone to have strong negative opinions about it)

runit is a fine init application. it might actually be better than OpenRC (which most of, our[0] systems run). Void is a fantastic distribution made better by a sane choice in init systems, and if you visit for that reason, you'll stay for all the other good, sane, choices and community.

0. http://medicalcannab.is

There is a lot of comments in the Devuan thread [1] and they will be here too, but we mostly skip over the actual problem systemd creates: very important parts of the Linux ecosystem becoming tightly coupled as to form a huge blob of programs where one either uses all or none. For one, I use Unix [2] as a productivity tool, not as a server, so I interact with init stuff very rarely. Thus I want a system that's easy to configure and debug. Systemd is the exact opposite of that, so I don't want to use it. But if I don't want to use it, now I have to find an alternative to or modify all the stuff that depends on it. If it was only an init script that I had to replace, that's okay, but there are tighter coupling between unrelated components and systemd. I've read that Gnome people consider having it as a dependency. Udev is so tightly coupled that Gentoo had to make a fork. But it should be enough to write a new startup script for switching to another init system, not fork the entire program I want to use. This is the problem with systemd.

[1] https://news.ycombinator.com/item?id=11601288

[2] Arch Linux for long, nowadays FreeBSD

And yet, it's not actually that hard to configure or debug. How is fiddling with symlinks and scripts easier than changing unit files?

With systemd you still have to fiddle with symlinks to enable/disable units (at least on the same level as other init/rc systems which have wrappers around ln).

And with a sane RC system, such as OpenRC (which technically is still SysV init, but who cares about the init process.. it doesn't do much really), I would argue the scripts are on the same level of complexity as systemd unit files for most cases (so it would be fine to use either).

But for more complex applications, such as mounting crypto devices or network configuration, these two do depart a fair bit. Systemd chose to not go the way of simply wrapping the cryptsetup command, but talk directly to the kernel, which would be fine or even better in theory, but has the slight drawback of reducing (as far as I could tell) all error conditions to Invalid Argument (which took me then quite a bit of effort to find out what I had done wrong). Systemd-network(d?) suffers from the same problem (doesn't use the iproute2 command, but directly speaks to the kernel) in my opinion - again it is only really useful for the most common use cases, uncommon ones become impossible (because certain iproute2 features are simply unsupported as of yet) or really difficult to debug (because you are debugging an application interpreting configuration files instead of simple shell scripts).

Hmmm... That's a fair point. I'm pretty certain you can replace the binaries with a script, really the value of units is to allow for parellilisation and dependency load order :-)

Both of which are also available in OpenRC (and other alternative systems such as runit, etc.) :)

No, runit cannot do that. You need to use OpenRC with runit.

Actually runit can do parallel startup (runsvdir tries to start all services in parallel) and can handle dependencies between services [1]. And there is at least one distribution that actually uses runit in this configuration [2].

[1] http://smarden.org/runit/dependencies.html [2] http://www.voidlinux.eu/usage/runit/

You might want to review https://news.ycombinator.com/item?id=11609291 .

I already replied to that before, but only now realized that both are the same author, which made this post about runit slightly redundant since I already touched on in my reply to that post.

Let me decide it.

If I don't want to use it, I should be able to swap it out. I shouldn't have to patch Gnome, udev, and whatnot to not use systemd.

I don't have problems with systemd, what I feat is essential parts of GNU and Linux infrastructure becoming directly dependent on it, at the actual software level.

It's way easier to depend on some functionality provided within the systemd project. Various bits do NOT require systemd to be the init system.

The conflict here is pretty easy:

  * People not wanting to avoid anything from systemd project
  * Maintainers simplifying their project by relying on an API

The API they're relying on is not init system specific, but because it is provided by the systemd project, there's non-technical hate against the decision. Outcome: maintainers ignore those people as they're irrational.

Have you considered using Knoppix? Or now Devuan? Because you don't have to use systemd...

Heck. I have to mount my USB sticks. I have to run some Gnome apps. What part of that you don't understand? I don't want to have to load my drivers manually just because I don't want to use systemd. Udev is the thing that probes devices at boot time, isn't it? I don't want to fork Emacs, on which my life depends, because dbus becomes a hard dependency, which becomes coupled to systemd. Are you paid to not understand?

At the end of the day, I've left GNU/Linux behind anyways, I'm not talking for myself but for the community.

Why would you need to do that? A change to the init system isn't going to break that. Are you actually using a system that has switched back to SysV scripts?!?

Avoiding the systemd thing here too, bsd for the win. Plenty has been said about it and some will never understand what is wrong with it.

Just curious: why does the rest of the world ignore things like launchd, libdispatch, etc?

People in FreeBSD-land have been look at launchd on and off:

https://wiki.freebsd.org/launchd

Launched was considered for a few moments by Ubuntu, but it wasn't a good fit and the license isn't deemed compatible with the licenses of the FSF.

Licensing, I'd guess. Also, Apple has no philosophical opposition to integrating things; UNIX people do.

Unix people? Not exactly the words i would have chosen, but i suppose you mean that linux has gpl licensing. BSD type licenses work well with unix,but are not compatible with the GPL.

That's simply untrue. First of all, you can run any program on top of Linux without licensing issues. Sure, Linux is GPL and so it much of the user land (all of GNU being a prime example). But if your program doesn't use any of those things outside of the "common use" of those components, the license is irrelevant. However, BSD-style licenses (excluding the now-unused 4 clause BSD) allow for sublicensing, so you can also include that code inside GPL projects (sublicensed under the GPL).

Having a standard that works across distros is a good thing, at least things don't feel as alien switching between redhat and debian (e.g. work and home).

Perhaps the title should have [2010] at the end.

Yeah, not much "rethinking" going on here anyway. Thought this was going to be something actually valuable not "doing exactly what PID 1 is already suppose to do"

Based on last time when someone decided to rethink PID1 and wrote systemd, I'm kind of afraid of any research in this field...

EDIT: flicked through the article and yup, it's 2010 and systemd is announced: "You probably guessed it: what I suggested above as requirements and features for an ideal init system is actually available now, in a (still experimental) init system called systemd, and which I hereby want to announce."

2016 and I'd still say it's mostly experimental, but unfortunately reached production machines now.

Did you not read the article? It's from 2010 and written by Lennart Poettering, introducing the basic concepts behind systemd.

Well I would say it's functionnal and hasn't reached enough production machines.

I don't understand the hate that systemd seems to bring in people. It may not be perfect but the good largely outweights the bad and anyway most of the arguments against it are the same copy pasted critics from day one, when they aren't pure name calling against its author.

From a user perspective I for one am glad to be able to start and kill services in a reliable maner, to have structured logging and a set of base system components that are simple to use, coherent and well documented and make taking advantage of otherwise painful to use Linux features easy.

Most anti-systemd people seem to be adverse to change and would rather stay on SysV init, but seriously how much longer are we expected to stick with that symlink mess of ugly scripts that can't reliably start a service in a reproducible environment or track which process belongs to which service ?

I often see anti-systemd people advocate a move to FreeBSD, but there is a reason behind the existence of the nosh or launchd projects... BSD init might be cleaner than SysV but it suffers from some of the same problems.

and anyway most of the arguments against it are the same copy pasted critics from day one

The age of an argument doesn't negate the argument, especially if it has not been effectively addressed since it was first proposed.

Those arguments are copy and pasted because they are still solid arguments and they have been pretty much ignored or handwaved away.

That you're tired of hearing them doesn't mean that they shouldn't be addressed. Those making the arguments are getting tired of having to repeat them—just like the tech industry is tired of having to continuously debate the government on cryptography: the arguments for liberty still stand, despite being codifed two centuries ago.

This seems like a pretty strong and specific technical argument in favor of systemd: https://lists.debian.org/debian-ctte/2013/12/msg00234.html

What's the best single page summary along with adequate details of the arguments against systemd, and what's the professed alternative? From Googling around, I haven't found anything with a comparable level of technical quality, depth, and concreteness.

Interesting link btw. I think the number of features is oversold a bit. Others here pointed out that, being so critical, it's best to have first process be simple as possible with another doing the complicated stuff. So, those features could be delegated into an isolated process via message passing.

However, it stood out that the shell scripts are complex and buggy in edge cases. Quite opposite of robust programming we normally want. The socket activation trick is nice. I could see following Systemd's lead to improve in those two areas with a regular, minimalist init. Other features in other processes. See where that goes before letting one component go octopus on everything else.

Btw, here's the answer to your recently deleted question:

http://www.throwww.com/a/7bn

Had to do some long shifts before digging it out. Not sure why you deleted it given it was a decent question. Still "self-censoring?" :P Anyway, you can email my profile address about it if still curious on technical angles and such.

Reply on news.ycombinator.com