A nice story about Unix processes “infecting” each other
rachelbythebay.com
rachelbythebay.com
I've seen quite a few people insist that SysV was perfectly fine and nothing about it needed fixing. But this article is a good example of how that kind of thing actually went: a myriad gotchas, bugs and undesirable characteristics eventually became so entrenched and familiar that knowing about all of this just became part of the sysadmin job.
Having an elaborate dance around system upgrades because it can break ssh was a perfectly normal thing. You're just supposed to know that.
And init scripts are fine as they are, you're just supposed to know processes inherit stuff, and take that into account.
And stray pid files sometimes stop a service from starting, that's also basic knowledge.
And rebooting? Nah, we'll keep the machine running for 2 years without a kernel upgrade, because who knows if all the services will come back after a reboot. Better not risk it.
And I kinda think I know why some people actually liked that. Figuring out problems like this when nobody else can and saving the day makes you feel like a wizard. The problem is that the competition didn't sit still and eventually Windows and Mac got good enough, and all this stuff became even more of a clunky impediment to getting stuff done.
I for one am very glad that systemd modernized things a bit and helped remove some of this stuff from the equation.
I, for one, I'm glad there are still options and that even Debian let the door open to, one day, switching to something else than systemd.
I'm running Devuan (Chimaera) now and it's fine (for my needs).
It's not all rosy: there's something to be said about the insane kitchen sink that systemd is. Having the PID1 so gigantic, with its tentacles everywhere in the system, ain't something the Linux community should be proud of. I'm very happy that, so far, Linus is resisting and nothing in the kernel is (yet?) tied to systemd.
Linux is about choices too. The day there ain't no choice anymore, I'm switching to a BSD.
I'm running Linus' Linux. Not IBM/Red Hat/Poettering's Linux.
Personally, I take the opposite stance. I have very little ideology besides things being open source. To me a machine is a tool to do a job, and if systemd does the job better, then that's the right tool, regardless of any violation of principles that it might involve. There's definitely good ideas in the Unix model, but if the model ceases to be useful, it's time to throw the bad parts out.
I also have no issues whatsoever with running Red Hat's Linux (I use Fedora). It does the job, it provides the tools I like, that's what matters.
The rest doesn't really matter. They're separate processes and many are task specific and not used if you don't need that particular bit of functionality.
You're right. It was an awesome intellectual high each time.
But only the first time. Eventually seeing the same stupid problem come back got tedious. Thank goodness our systems continue to evolve. It's so long since I've seen a leaky environment at service startup cause problems that I'd forgotten it was a thing.
I don't understand anyone that seems to think UNIX was perfected in 1983 with the release of System V. We've learned so much since then by standing on the shoulders of giants.
More to the point is how Linux community tends to re-invent poorly or be against stuff that had been already available for ages on commercial UNIXes, the ones that I had more fun using actually.
The general concept needs little more than a single dependency graph, and yet they've created like 3+. Services are enabled/disabled by a separate graph of Wants/WantedBy, plus a whole bunch of ad-hoc look-them-up-every-time PartOf/ReloadPropagatedFrom/etc. It's like they did the brainstorming design phase, but never did the design optimization bit where you try to simplify the model.
Much of the pain of systemd goes away under NixOS, with the functional/exit (as contrasted with mutable/voice) thing of hiding all the tedious stateful complexity behind persistent objects, and leaving you with just an overly ugly config format. But then you've got to ask why the systemd designers couldn't have created that interface in the first place! One file per service config and one (conceptual) top level file for the dependency graph! Then the traditional distros could manage the complexity of needing to reference every possible service package with their usual include .d schemes, and anybody breaking from that could just augment or fork the toplevel config as usual.
/lib/systemd and /etc/systemd make perfect sense to me -- /lib/systemd is what the package installs. /etc/systemd is the admin overriding what's in /lib. That's a perfectly fine solution in that the admin's modifications apply cleanly on top of whatever the package installs, and works nicely with package updates. Installing files into /etc is troublesome.
Stuff like PropagatesReloadTo exists because there exist things that are built from multiple cooperating units that need such measures. The vast majority of things don't need anything more than a 5 line service file.
One file for the dependency graph? How would that work, and why would that be a good idea? In systemd, the service declares what it needs. Why would I want to manually go mess with a file after installing some third party service to tell the system what that service needs?
Installing default files into /etc is what every Debian service does. When the user edits them, they take over responsibility. Same as installing them in /lib and then manually dropping an override in /etc, only with all files (default and edited) in the same place.
> Stuff like PropagatesReloadTo exists because there exist things that are built from multiple cooperating units that need such measures
Every little feature is obviously needed for some purpose. The problem is they never did the design step to whittle down the model such that all those extra facets were subsumed by fewer more general features.
> Why would I want to manually go mess with a file after installing some third party service to tell the system what that service needs?
A distro would have the top-level file defining the graph of all the distro packages you could possibly install, so you would not need to edit any files.
Yeah, dealt with that, it sucks. You upgrade, and then end up staring at half a dozen 3-way diffs for services you're only vaguely aware exist. Sometimes the packagers modify the comments or rearrange the file, so figuring out whether anything of importance happened can be tricky.
Systemd makes this nice and easy. If I add an environment variable to a service, it stays added after an upgrade without any fuss. I can look in /etc and see exactly what was overriden, and trivially return to an unmodified state if needed.
> The problem is they never did the design step to whittle down the model such that all those extra facets were subsumed by fewer more general features.
I'm not sure how could that be done without going to every software maker out there and telling them to ensure such a feature wouldn't be needed. I'm not seeing that happening.
> A distro would have the top-level file defining the graph of all the distro packages you could possibly install, so you would not need to edit any files.
Lots of stuff doesn't come with distros, including the packages for my own project which ships systemd units. Add a third party repo, install packages, enable services, done. No need to mess around with some global dependency config.
This is a fundamental problem that you cannot escape. If you take something from upstream and modify it, then you're stuck keeping up with the changes. If you're only "vaguely aware" of something you modified, it's because you forgot.
> Systemd makes this nice and easy
No, it doesn't. You're describing doing the exact same thing as if you just ignore the config file diffs - keeping your own version and ignoring upstream. If the upstream changed something relevant, your forked config breaks.
> I'm not sure how could that be done without going to every software maker out there and telling them to ensure such a feature wouldn't be needed.
I'm talking about the design of systemd, not the complexity of services. We're not going to be able to hash out how they could simplify their own overall model in a message board thread, but suffice it to say that 11 different types of dependencies (double including their reversals) could certainly be pared down.
> Lots of stuff doesn't come with distros, including the packages for my own project which ships systemd units. Add a third party repo, install packages, enable services, done
Your packages would drop a unit in say /etc/sysd/units/, and augment the toplevel config by dropping something in say /etc/sysd/graph.d/
You can escape it if you design for it. You can't escape it for exim.conf, but you can escape it for systemd's own files, because it designed for that particular case, as explained below.
> No, it doesn't. You're describing doing the exact same thing as if you just ignore the config file diffs - keeping your own version and ignoring upstream. If the upstream changed something relevant, your forked config breaks.
No. The way it works is that stuff in /etc overrides stuff in /lib, line by line. Eg, in /etc you place something like this:
[Service]
Environment=FOO=bar
> I'm talking about the design of systemd, not the complexity of services.
One and the same. Systemd exists to run existing services. If an existing program is composed of 3 cooperating binaries and needed hacks where signals must be sent to the other processes, there's nothing for systemd to do but to implement the required functionality to support that.
> Your packages would drop a unit in say /etc/sysd/units/, and augment the toplevel config by dropping something in say /etc/sysd/graph.d
I'm not seeing the benefit then. What's the difference, keeping the config in two files vs one?
Oof, I did not know that. Likely because the last thing I'd want to do is split a simple service config between an even larger number of files. Also IME automerging just ends up creating a different type of upgrade error - it's more straightforward to just fork the whole config and take over responsibility. Software packages generally don't remove functionality, but default config files often change as maintainers get better ideas.
> there's nothing for systemd to do but to implement the required functionality to support that
Eleven types of dependencies. I know many of those could be implemented more generally. Perhaps not straightforwardly in an ini file, but that's what I mean about overall design.
> What's the difference, keeping the config in two files vs one?
A single service config is currently kept in at least 4 config files - the lib unit, the etc unit, the wants symlink, the wantedby symlink. Never mind all the extra .target files and their own duplications, plus the reverse injections. And this is for every single service. In my example, the distribution services would each only have one file, as their dependencies would be in an easily readable graph file. The answer to the expression problem is to add the appropriate hooks, not to just give up on factoring.
The standard response from systemd advocates about all this complexity is to use bespoke tools to analyze its computed internal model. This would be reasonable if it didn't purport to be working with the configuration file model. Instead it's like a bait and switch of complexity. It would have been more honest to store the config in sqlite.
That's one reason why the systemd unit files are declarative -- things are unlikely to go wrong, because declarations are easy to override or add to without things going horribly wrong as a result.
And 99% of the time, you're not doing this at all and just using the defaults.
> Eleven types of dependencies. I know many of those could be implemented more generally.
Most of those are kind of special cases you don't normally use, but that are needed for some obscure application or two.
> A single service config is currently kept in at least 4 config files - the lib unit, the etc unit, the wants symlink, the wantedby symlink. Never mind all the extra .target files and their own duplications, plus the reverse injections. And this is for every single service
No, not really.
The lib unit exists always. That's what defines what the unit is, what it wants, etc. That's the only thing you ship for most things.
The etc unit exists only if the administrator didn't like something about the system one and decided to modify something. It's not part of a package, the admin creates it.
The symlinks are system configuration state -- that's what "systemctl enable" does. They're not part of the unit, they tell the system what to start on boot. The admin creates them when enabling a service.
.target units are the equivalent of runlevels. They're a way to bring the whole system into a desired state easily. They're mostly a distro-level thing and are shipped with it.
> The standard response from systemd advocates about all this complexity is to use bespoke tools to analyze its computed internal model. This would be reasonable if it didn't purport to be working with the configuration file model. Instead it's like a bait and switch of complexity. It would have been more honest to store the config in sqlite.
I don't quite understand. You seem to be confusing the difference between what the service itself declares ("I need a network connection") and the system's configuration state ("Apache should be started on boot").
You're pulling out the three dog defense here. Yes, unnecessary config complexity will make it so you have more conflicts. But simplification cannot eliminate the problem of merging. And we're explicitly talking about the times you need to interact with the system on this level.
> No, not really. [explanation of 4 different files]
Yes, really. You need to look at all of those to know a unit definition. And then you need to look at those for all the other units to see the dependency graph. Except nobody is capable of holding all that in their head. The closest programming analogy is Java, which effectively requires using an IDE to manage the complexity of splaying out structure in lots of tiny files.
> You seem to be confusing the difference between what the service itself declares and the system's configuration state
When you're the sysadmin writing/modifying a service, which is the only time you're touching unit files, the abstract distinction becomes moot - there is little point to installing a new unit without also enabling it. Meanwhile WantedBy etc can very much exist in unit files and not just the symlinks, and therefore to grok the complexity you need to take it into account.
As I referenced, take a look at how NixOS uses systemd while encapsulating its filesystem mess. There's no concept of a disabled unit - you take it out of the config and the unit file disappears. There's no concept of overriding a config, since that's done at the level of system config. There's no concept of making mutable changes to the system state, since you just rebuild a whole new config. If you do fark up, make a loop, and your system doesn't boot, you don't diagnose where that happened but just boot the independent old configuration without the loop. Whereas on Debian you're suffering the mental overhead from these needless reinventions.
The what now?
> Yes, unnecessary config complexity will make it so you have more conflicts. But simplification cannot eliminate the problem of merging.
Ok, do tell me how the example I posted could break. I'm curious.
> Yes, really. You need to look at all of those to know a unit definition.
No, the definition is in the .service file. There's a handy 'systemctl cat' command which will dump the full definition to the console, with any overrides included.
Which is how you actually do these things, btw. You're not supposed to poke around in the files and form a "mental map". You ask the system what you want to know. If you want to know what sshd needs to start, you do `systemctl list-dependencies sshd`. If you want to know what needs sshd, then `systemctl list-dependencies --reverse sshd`. The info comes out clearly, and nicely formatted. You can even graphically plot the boot process if you want to.
> When you're the sysadmin writing/modifying a service, which is the only time you're touching unit files, the abstract distinction becomes moot - there is little point to installing a new unit without also enabling it.
There certainly is a point. Some services can't be started without configuring them and have no possible useful defaults. Eg, anything that needs an external database. Or stuff that needs security considerations. Or stuff where there's optional functionality in a package, such as that you get smartctl and smartd in the same package. You might want to use the commandline tool without setting up monitoring.
> If you do fark up, make a loop, and your system doesn't boot, you don't diagnose where that happened but just boot the independent old configuration without the loop.
Why would you want a system where you can even make a loop? It should be impossible for it to happen in the first place.
Upstream starts off not setting FOO. You set FOO=bar. Upstream then sets FOO=baz and FOO_OPT=qux. Now the calculated config sees FOO=bar and FOO_OPT=qux, which was nobody's intention and is likely inconsistent.
> There's a handy 'systemctl cat' command which will dump the full definition to the console, with any overrides included... Which is how you actually do these things, btw. You're not supposed to poke around in the files and form a "mental map". You ask the system what you want to know.
I explicitly acknowledged this as part of my argument two comments back.
> Some services can't be started without configuring them and have no possible useful defaults
If a sysadmin is writing a unit file, it's because they're creating a new service to run. The generality of writing a unit file that an end user might want to run has dropped away.
> Why would you want a system where you can even make a loop? It should be impossible for it to happen in the first place.
Exactly! I'm talking about my own experience with systemd. I created a dependency loop with some ancillary high-level service, systemd responded by choosing an arbitrary place to break the cycle, and I was left with a completely unbootable system. (preemptively: I'm sure there is some bespoke command I could have run to shake that out before rebooting, but that flows into my larger argument)
Possible issue, yes. If this is an intended software setting, then I'd expect the software to take such a possibility into account, and log something useful. If the admin is screwing around with something like LD_PRELOAD, that's on the admin. Still, you have a file with nothing but what the admin changes. Easy to notice and undo. You don't have to diff the file against the upstream.
> If a sysadmin is writing a unit file, it's because they're creating a new service to run.
Sysadmins don't write unit files most of the time, they're distro provided. Think here of something like gitea -- it can be provided by the distro, but it doesn't work unless configured first. Where are your git repos? Which database are you going to use? Where's the mail server? All of that needs to be set after installing the package and it makes no sense to try starting it until that's done. So the sane initial state for it is "installed, but not started on boot".
> Exactly! I'm talking about my own experience with systemd. I created a dependency loop with some ancillary high-level service, systemd responded by choosing an arbitrary place to break the cycle, and I was left with a completely unbootable system.
Well, that sounds not good, and I'd say there should be some sort of sanity check for that.
Well, I haven't changed jobs in about 5 years - but there are plenty of indicators of the same problem around.
Less the, 'we never reboot' thing but more... 'that person is the only one who understands this'; leaving the problem ripe for creation on any given day.
Also a very large business/conglomerate - it's a household name.
I'm part of the newer generation in a sense - I picked up Linux and the like in the mid-early 2000s. I was a glutton for punishment. I wanted to see everything break however it could.
Eventually I learned quite a bit of these lessons out of sheer interest. It provides a lot of opportunity for the 'wizard' moments OP mentioned.
I find a fairly cavalier attitude about things as a result. People at work think I'm crazy, but the longer this is prolonged - the more likely it is to hurt.
> who knows if all the services will come back after a reboot. Better not risk it.
This is not a story about how bad sysV was but about how people happily disregarded any sort of infrastructure rigor "in the good old days". Applying upgrades that would break their system then wrote workarounds on top of others for those who also didn't care to test the builds, was considered reasonable. This isn't considered a good plan today and it wasn't a good plan then.
In short, yeah, life before systemd was hard.
Two somewhat topical observations in a similar vein:
Back in a sysvinit world, many (Debian) users would run init-scripts directly via `/etc/init.d/someservice someaction`, instead of using `/usr/sbin/service someservice someaction`. This could have very unpleasant side effects at times, because `service` is/was actively trying to clean and sanitize the resulting child processes' environment, while most init scripts themselves usually did not.
The second thing that the article made me think of is not directly related to most "admin wants the system to start a service"-scenarios, but COULD also happen in such a situation: On Linux, when a process tries to fork, the kernel requires certain guarantees in terms of usable memory for the process' copy, due to copy-on-write semantics for allocated pages. If these cannot be matched, forking might fail with ENOMEM. In my experience, this is much more likely to happen when the process trying to fork has a lot of allocated memory accounted for. Therefore, it makes perfect sense to have a "lightweight" kind of fork-and-exec-server around in the system that can be contacted via an IPC mechanism, and made to start daemon processes that are probably not meant to inherit anything from the caller's environment (like file descriptors, niceness, environment variables) in the first place.
The problem turned out to be that one sysadmin thought that the printer subsystem needed to be restarted after adding a printer queue, and to do it they used `sudo su` (not `sudo su -`) to gain privileges, and they liked a BSD-ish environment, so they had `/usr/ucb` ahead of `/usr/bin` (yes, this was on Solaris).
How would that break printing? Well, those HP Jetadmin printer drivers were awful Bourne shell scripts that depended on System V echo behavior to emit escape sequences for PCL, and now you see the problem: /usr/ucb/echo did not do what /usr/bin/echo did, and failed to emit those escape sequences, leading to garbage rather than properly-formed PCL being sent to the printers.
SMF (Solaris/Illumos) / systemd (Linux) fix the problem permanently.
Sometimes it's nice to look back at the fun bugs we squashed, and be grateful they're not a thing anymore because our systems have improved.
So much of the "Unix processes" theory. Given systemd did not stop to use "Unix processes", that should also have been a clue.
The issue was about the postinst script which either missed that it should have done something differently, or is maybe just not designed to be run nicely so the users broke an (implied) contract by calling it that way.
And since a lot of stuff is on lower ports, and there wasn't the complicated firewalls of today to dynamically flick around port routing depending on processes and the such, most programs grabbed the requested ports, and then setuid to the relevant daemon user, all by themselves (also forking, going into background, etc)
Thankfully these days its all unneeded, yes
TOPS-10, DEC's main operating system for PDP-10, had an interesting approach to this. The login system call, which was used to set the user ID for a user, was not privileged. However, it would return an error if it was executed when already logged in.
As was common with systems back then, there was a CLI that was built in to the OS and ran on all terminals without being logged in. It allowed you to run a restricted set of programs without logging in, including the login program.
One of those programs you could run without being logged in was queue, which was a program to view the print queue (and maybe other queues, like the queue of pending tape mounting requests?...I don't remember).
Unlike the login program, the queue program was also useful when logged in and so included many features and options beyond just what was needed by people who were not logged in. That included a flag to run other programs: queue/run:foo would run run foo.exe. (No, I don't remember why /run was ever actually useful).
Someone figured out that this allowed queue/run:ddt to run the DDT debugger, and that they could do this while not logged in thus ending up with a running DDT. They could then use the debugger to poke in a short assembly program to invoke the login system call and login as anyone they wanted, no password required.