why the aversion to systemd?
I don't like everything depending on systemd it feels like too much complexity at the wrong part of the stack. It doesn't jive with my sense of architecture, it's not well designed software.
Basically, a task running as pid 1 (including in a container) could call a new kpid1() syscall, which would cause the kernel to completely take it over. The kernel would take care of all the usual init work, and it would expose the minimal API (presumably using a new kind of fd) to allow a different task to give it instructions and manage zombies as needed. And that’s it.
It’s worth noting that the entire concept of pid 1 is very unixy, but not in a good way. Reasonable modern designs (e.g. all the Windows variants) don’t have any real equivalent.
If you take anything resembling a fresh look at this concept, it's absurd. Imagine if every open file had a systemwide unique id, and one specific process owned that id and would continue to own it until it released it.
Reasonable designs use weak references that don't have values that can be compared across processes. These are usually called "handles" or "file descriptors", and they don't have this problem at all. Nothing reaps sockets, for example, and nothing needs to.
Zombie reaping could have a reasonable API. (Signals are miserable.)
PID 1 is magic in problematic ways. In particular, if PID 1 crashes, the whole system goes down with it. And having PID 1 be a normal program running from a normal ELF file means that that ELF file is pinned for the life of the system or at least until it execs something else. So handoff from initramfs to a real fs either involves PID 1 calling execve() or involves leaving the init process around. Upgrading the package containing PID 1 requires execve(). Running PID 1 from a network filesystem or an unreliable device risks a kernel panic for no good reason.
With PID 1 moved to the kernel, the actual service management job is no longer coupled to PID 1’s legacy. A service manager could hand off to another one by saving its state to disk and exiting, by running the new one and moving its state after the new one starts, or by any other ordinary means. And if it crashes, you can ssh in, read logs, save work, and then restart it or the whole system as appropriate.
As a minor additional benefit, having PID 1 in the kernel could enable some optimizations. Right now, a process must enter the zombie state when it exits, and it must stay in that state until its parent wakes up and reaps it. So a service exiting fundamentally involves some complex bookkeeping and a context switch to a single, unrelated process. If the kernel knew that kpid1 was in use and that nothing in the system actually needs to be notified of exiting children of pid 1, then a child of pid 1 that exits could simply go away, as it would on a sensible system like Windows.
(Yes, it's okay to admit that, in some respects, Windows is substantially better than Linux/Unix.)
> text files or symlinks for configuration and text files for logging.
Systemd gets this right and arguably pushed the whole ecosystem in this direction. The old rc scripts could barely be considered a text file and symlink configuration system — they were a pile of text files containing a miserable combination of code and configuration mixed together, along with a very simple configuration (this service is enabled in these runlevels, more or less) that got translated, hopefully correctly, into symlinks. Of course, nothing really kept the symlink farm consistent with itself or anything else except a pile of additional scripts associated with packages that tried and usually succeeded.
I'm not sure what the "right" solution looks like, perhaps a directory of TOML or JSON files. Perhaps the aforementioned + executable shell scripts with predictable naming? handwave handwave as long as it's "UNIXy". (consist of easy to edit text & not invent a new anything & be composed of pieces which do one thing well)
> text files for logging
...just sucks, on both embedded systems and production servers. (I.e. anywhere where you aren't debugging the machine on the machine, but rather from another machine.)
Either the program just writes to a plain text file forever — and so fills up your disk the first time it goes haywire (so now you have two critical runtime problems!); or it implements its own log rotation and compression (as must every other daemon — not very unixy!); or it must be specifically wired to work with syslog APIs in order to use rsyslog (which, by the way, uses binary wire protocols as well; logging at scale hasn't been text-based in a long time.)
Journald, meanwhile, just sits on the other side of the pipe from any systemd service-unit's stdout + stderr; manages log rotation + compression in a centralized way (which also means you get cross-unit log compression for free); and offers CLI tooling to pipe the multiplexed log stream back into anything that wants to read from it, in whatever format those things want to read from it (i.e. tools that want JSON Lines, get JSON Lines; tools that want plaintext, get plaintext; tools that want a binary record stream, get a binary record stream.)
Is this a Unixy approach? Well, it's pretty much the same one taken by the extremely venerable Unix/Linux line-printer (lp) subsystem — CLI commands, with textual config files, for interacting with a system daemon (lpd) that manages and manipulates binary state files, within daemon-owned directories. Would you complain that the contents of /var/spool/lpd aren't human-readable?
Ehrm, no. Managing a sizeable fleet, with a central logging server for 1.5 decades, and we never had the problems you mentioned:
> and so fills up your disk the first time it goes haywire.
This is a bug of the program or configuration mistake or your monitoring is not working as intended.
Funnily, we're seeing more disk pressure from systemd journals. Go, figure.
Just remembered: syslog daemons have rate-suppression mechanisms to prevent big lines repeating too fast and preventing your disk from filling up. So even your program enters an infinite loop, a well configured syslog daemon (rsyslog, syslog-ng, whatnot), should note "X similar errors have been supressed.", where X can be anything from 2 to 1000 (or even more).
> or it implements its own log rotation and compression
Which you can disable 99% of the time and just delegate the stuff to logrotate.
> it must be specifically wired to work with syslog APIs in order to use rsyslog
rsyslog is just a syslog daemon. syslog is kernel plumbing at this level. You can terminate this pipe with anything.
> Journald, meanwhile, just sits on the other side of the pipe from any systemd service-unit's stdout + stderr; manages log rotation...
And provides nothing new when compared to syslog plumbing. A binary log, some tooling around that, and that's it. It even makes per daemon log monitoring harder by blinding syslog-aware monitoring and automation tools, hence we need to enable rsyslog on the system too. Now we have two journals. Neat.
> venerable Unix/Linux line-printer (lp) subsystem
Which only handles "line-printer" subsystem, and yes, it's more UNIXy. It doesn't get the text output, bashes into a binary data structure, and doesn't try to replace anything and everything from boot to logs to time sync to user login to tap water temperature.
It just stores its state in binary file. Which almost every UNIX daemon does. Incl, but not limited to X11 & CUPS.
So configure journald correctly? It has multiple options to control disk usage from logs -- `man journald.conf` and search for "MaxUse" for the relevant options.
> Just remembered: syslog daemons have rate-suppression mechanisms
So does journald. Relevant options are RateLimitIntervalSec and RateLimitBurst, and individual services can set their own limits as well.
So configure logrotate correctly; you hardly need journald for that.
Like...Systemd?
How is that different from I don't like relativity theory at a fundamental level. It feels abundant, bloated, and not very physical. I'd rather have my Newton's law with a handful of augmentation.
I fought against systemd for a while, too
I was wrong
The "if it ain't broke, why fix it" approach with 'classic' init scripts led to a far far messier place than systemd
My only complaint about systemd is that I haven't found a way to push the journal to text simply ...and that's most likely my poor google-fu or not having enough to time to fully dive into it, rather than "systemd's journal sucks"
Systemd does use "text files or symlinks for configuration"
It's very well designed - though you may happen to have a preference for a different architecture (...but the one you described is pretty much systemd, my friend :))
Would you write an optimizing compiler in multiple small tools as well? Essential complexity can’t be reduced, IPC will blow up your accidental complexity budget. Many times a monolith is indeed the best design choice.
Now it's better, not but perfect, and a lead dev working for Microsoft doesn't inspire confidence about its long term agenda.
I can post links to comments of mine if you're interested further.
I’d like to learn more!
My issues with it are:
- If you setup an atypical configuration, particularly involving luks volumes, it is not hard to break systemd and dracut's assumptions, and then you will have a hell of time trying to boot and survive systemd updates.
- When it breaks, figuring out what the hell is going on involves having to learn a lot of systemd, which has lots of its own unique vocabulary and logic. There are many pieces and moving parts. It feels like someone went "microservice-crazy" with the init system and like there has to be a simpler way. The surface area of systemd is enormous.
- The whole anti-split /usr crusade is excessive. You might want to have /usr as a separate volume so you can mount it with the nodev option, for example. Why should that be forbidden?
If you conform to systemd's expectations about system configuration, I'm sure it works fine regardless of its elegance/inelegance and excessive complexity. If you would like to do things differently in ways that Linux's building blocks otherwise permit, you could be in serious trouble.
:P
For myself, eh. I find it a little annoying but basically tolerable, sort of like a reinvention of SMF from solaris. Linux system config / init has gotten a hell of a lot more complex since I first touched it in the mid-90s, sometimes we get more functionality for that and other times the grognard in me wants to bin it all and retreat to Slackware or something. What was the old joke, Microsoft admins have solitare.exe and Linux admins have "fiddling with text files"? :)