Hacker News Reader

> Figuring out problems like this when nobody else can and saving the day makes you feel like a wizard.

You're right. It was an awesome intellectual high each time.

But only the first time. Eventually seeing the same stupid problem come back got tedious. Thank goodness our systems continue to evolve. It's so long since I've seen a leaky environment at service startup cause problems that I'd forgotten it was a thing.

I don't understand anyone that seems to think UNIX was perfected in 1983 with the release of System V. We've learned so much since then by standing on the shoulders of giants.

Somehow there is this mysticism that UNIX System V was the best thing since sliced bread, only outperformed by Plan 9.

More to the point is how Linux community tends to re-invent poorly or be against stuff that had been already available for ages on commercial UNIXes, the ones that I had more fun using actually.

I did some contract work for a company that hadn't rebooted a couple of servers in almost 5 years because they were afraid of them not coming back up. Apparently the only employee who knew anything about that system left. This was an organization bringing in 100's of millions in revenue, not a tiny company.

This is a fun self-realizing thing, and I see it all over the place even today!

Well, I haven't changed jobs in about 5 years - but there are plenty of indicators of the same problem around.

Less the, 'we never reboot' thing but more... 'that person is the only one who understands this'; leaving the problem ripe for creation on any given day.

Also a very large business/conglomerate - it's a household name.

I'm part of the newer generation in a sense - I picked up Linux and the like in the mid-early 2000s. I was a glutton for punishment. I wanted to see everything break however it could.

Eventually I learned quite a bit of these lessons out of sheer interest. It provides a lot of opportunity for the 'wizard' moments OP mentioned.

I find a fairly cavalier attitude about things as a result. People at work think I'm crazy, but the longer this is prolonged - the more likely it is to hurt.

The problem with systemd isn't the modernization, it's the utterly ungrokkable bloat caused by not making a clean break from many old customs. Countless .ini files splayed across /lib/systemd and /etc/systemd with symlinks setting additional properties? It (ab)uses the filesystem to create the illusion of being able to grok it with standard file manipulation tools, despite that being impossible.

The general concept needs little more than a single dependency graph, and yet they've created like 3+. Services are enabled/disabled by a separate graph of Wants/WantedBy, plus a whole bunch of ad-hoc look-them-up-every-time PartOf/ReloadPropagatedFrom/etc. It's like they did the brainstorming design phase, but never did the design optimization bit where you try to simplify the model.

Much of the pain of systemd goes away under NixOS, with the functional/exit (as contrasted with mutable/voice) thing of hiding all the tedious stateful complexity behind persistent objects, and leaving you with just an overly ugly config format. But then you've got to ask why the systemd designers couldn't have created that interface in the first place! One file per service config and one (conceptual) top level file for the dependency graph! Then the traditional distros could manage the complexity of needing to reference every possible service package with their usual include .d schemes, and anybody breaking from that could just augment or fork the toplevel config as usual.

I'm not familiar with NixOS, so can't argue there. However:

/lib/systemd and /etc/systemd make perfect sense to me -- /lib/systemd is what the package installs. /etc/systemd is the admin overriding what's in /lib. That's a perfectly fine solution in that the admin's modifications apply cleanly on top of whatever the package installs, and works nicely with package updates. Installing files into /etc is troublesome.

Stuff like PropagatesReloadTo exists because there exist things that are built from multiple cooperating units that need such measures. The vast majority of things don't need anything more than a 5 line service file.

One file for the dependency graph? How would that work, and why would that be a good idea? In systemd, the service declares what it needs. Why would I want to manually go mess with a file after installing some third party service to tell the system what that service needs?

> Installing files into /etc is troublesome

Installing default files into /etc is what every Debian service does. When the user edits them, they take over responsibility. Same as installing them in /lib and then manually dropping an override in /etc, only with all files (default and edited) in the same place.

> Stuff like PropagatesReloadTo exists because there exist things that are built from multiple cooperating units that need such measures

Every little feature is obviously needed for some purpose. The problem is they never did the design step to whittle down the model such that all those extra facets were subsumed by fewer more general features.

> Why would I want to manually go mess with a file after installing some third party service to tell the system what that service needs?

A distro would have the top-level file defining the graph of all the distro packages you could possibly install, so you would not need to edit any files.

> Installing default files into /etc is what every Debian service does. When the user edits them, they take over responsibility.

Yeah, dealt with that, it sucks. You upgrade, and then end up staring at half a dozen 3-way diffs for services you're only vaguely aware exist. Sometimes the packagers modify the comments or rearrange the file, so figuring out whether anything of importance happened can be tricky.

Systemd makes this nice and easy. If I add an environment variable to a service, it stays added after an upgrade without any fuss. I can look in /etc and see exactly what was overriden, and trivially return to an unmodified state if needed.

> The problem is they never did the design step to whittle down the model such that all those extra facets were subsumed by fewer more general features.

I'm not sure how could that be done without going to every software maker out there and telling them to ensure such a feature wouldn't be needed. I'm not seeing that happening.

> A distro would have the top-level file defining the graph of all the distro packages you could possibly install, so you would not need to edit any files.

Lots of stuff doesn't come with distros, including the packages for my own project which ships systemd units. Add a third party repo, install packages, enable services, done. No need to mess around with some global dependency config.

> You upgrade, and then end up staring at half a dozen 3-way diffs for services you're only vaguely aware exist

This is a fundamental problem that you cannot escape. If you take something from upstream and modify it, then you're stuck keeping up with the changes. If you're only "vaguely aware" of something you modified, it's because you forgot.

> Systemd makes this nice and easy

No, it doesn't. You're describing doing the exact same thing as if you just ignore the config file diffs - keeping your own version and ignoring upstream. If the upstream changed something relevant, your forked config breaks.

> I'm not sure how could that be done without going to every software maker out there and telling them to ensure such a feature wouldn't be needed.

I'm talking about the design of systemd, not the complexity of services. We're not going to be able to hash out how they could simplify their own overall model in a message board thread, but suffice it to say that 11 different types of dependencies (double including their reversals) could certainly be pared down.

> Lots of stuff doesn't come with distros, including the packages for my own project which ships systemd units. Add a third party repo, install packages, enable services, done

Your packages would drop a unit in say /etc/sysd/units/, and augment the toplevel config by dropping something in say /etc/sysd/graph.d/

> This is a fundamental problem that you cannot escape.

You can escape it if you design for it. You can't escape it for exim.conf, but you can escape it for systemd's own files, because it designed for that particular case, as explained below.

> No, it doesn't. You're describing doing the exact same thing as if you just ignore the config file diffs - keeping your own version and ignoring upstream. If the upstream changed something relevant, your forked config breaks.

No. The way it works is that stuff in /etc overrides stuff in /lib, line by line. Eg, in /etc you place something like this:

    [Service]
    Environment=FOO=bar

And that's all the file contains. This file adds an extra environment variable to the unit it overrides, and nothing more. If upstream has two variables now and adds a third in the next release, then it all still adds up correctly and without any interaction. There's also a special syntax for saying "Ignore whatever this says in the packaged original, I only want my version".

> I'm talking about the design of systemd, not the complexity of services.

One and the same. Systemd exists to run existing services. If an existing program is composed of 3 cooperating binaries and needed hacks where signals must be sent to the other processes, there's nothing for systemd to do but to implement the required functionality to support that.

> Your packages would drop a unit in say /etc/sysd/units/, and augment the toplevel config by dropping something in say /etc/sysd/graph.d

I'm not seeing the benefit then. What's the difference, keeping the config in two files vs one?

> The way it works is that stuff in /etc overrides stuff in /lib, line by line

Oof, I did not know that. Likely because the last thing I'd want to do is split a simple service config between an even larger number of files. Also IME automerging just ends up creating a different type of upgrade error - it's more straightforward to just fork the whole config and take over responsibility. Software packages generally don't remove functionality, but default config files often change as maintainers get better ideas.

> there's nothing for systemd to do but to implement the required functionality to support that

Eleven types of dependencies. I know many of those could be implemented more generally. Perhaps not straightforwardly in an ini file, but that's what I mean about overall design.

> What's the difference, keeping the config in two files vs one?

A single service config is currently kept in at least 4 config files - the lib unit, the etc unit, the wants symlink, the wantedby symlink. Never mind all the extra .target files and their own duplications, plus the reverse injections. And this is for every single service. In my example, the distribution services would each only have one file, as their dependencies would be in an easily readable graph file. The answer to the expression problem is to add the appropriate hooks, not to just give up on factoring.

The standard response from systemd advocates about all this complexity is to use bespoke tools to analyze its computed internal model. This would be reasonable if it didn't purport to be working with the configuration file model. Instead it's like a bait and switch of complexity. It would have been more honest to store the config in sqlite.

> Oof, I did not know that. Likely because the last thing I'd want to do is split a simple service config between an even larger number of files. Also IME automerging just ends up creating a different type of upgrade error - it's more straightforward to just fork the whole config and take over responsibility.

That's one reason why the systemd unit files are declarative -- things are unlikely to go wrong, because declarations are easy to override or add to without things going horribly wrong as a result.

And 99% of the time, you're not doing this at all and just using the defaults.

> Eleven types of dependencies. I know many of those could be implemented more generally.

Most of those are kind of special cases you don't normally use, but that are needed for some obscure application or two.

> A single service config is currently kept in at least 4 config files - the lib unit, the etc unit, the wants symlink, the wantedby symlink. Never mind all the extra .target files and their own duplications, plus the reverse injections. And this is for every single service

No, not really.

The lib unit exists always. That's what defines what the unit is, what it wants, etc. That's the only thing you ship for most things.

The etc unit exists only if the administrator didn't like something about the system one and decided to modify something. It's not part of a package, the admin creates it.

The symlinks are system configuration state -- that's what "systemctl enable" does. They're not part of the unit, they tell the system what to start on boot. The admin creates them when enabling a service.

.target units are the equivalent of runlevels. They're a way to bring the whole system into a desired state easily. They're mostly a distro-level thing and are shipped with it.

> The standard response from systemd advocates about all this complexity is to use bespoke tools to analyze its computed internal model. This would be reasonable if it didn't purport to be working with the configuration file model. Instead it's like a bait and switch of complexity. It would have been more honest to store the config in sqlite.

I don't quite understand. You seem to be confusing the difference between what the service itself declares ("I need a network connection") and the system's configuration state ("Apache should be started on boot").

> That's one reason why the systemd unit files are declarative ... 99% of the time, you're not doing this at all

You're pulling out the three dog defense here. Yes, unnecessary config complexity will make it so you have more conflicts. But simplification cannot eliminate the problem of merging. And we're explicitly talking about the times you need to interact with the system on this level.

> No, not really. [explanation of 4 different files]

Yes, really. You need to look at all of those to know a unit definition. And then you need to look at those for all the other units to see the dependency graph. Except nobody is capable of holding all that in their head. The closest programming analogy is Java, which effectively requires using an IDE to manage the complexity of splaying out structure in lots of tiny files.

> You seem to be confusing the difference between what the service itself declares and the system's configuration state

When you're the sysadmin writing/modifying a service, which is the only time you're touching unit files, the abstract distinction becomes moot - there is little point to installing a new unit without also enabling it. Meanwhile WantedBy etc can very much exist in unit files and not just the symlinks, and therefore to grok the complexity you need to take it into account.

As I referenced, take a look at how NixOS uses systemd while encapsulating its filesystem mess. There's no concept of a disabled unit - you take it out of the config and the unit file disappears. There's no concept of overriding a config, since that's done at the level of system config. There's no concept of making mutable changes to the system state, since you just rebuild a whole new config. If you do fark up, make a loop, and your system doesn't boot, you don't diagnose where that happened but just boot the independent old configuration without the loop. Whereas on Debian you're suffering the mental overhead from these needless reinventions.

> You're pulling out the three dog defense here.

The what now?

> Yes, unnecessary config complexity will make it so you have more conflicts. But simplification cannot eliminate the problem of merging.

Ok, do tell me how the example I posted could break. I'm curious.

> Yes, really. You need to look at all of those to know a unit definition.

No, the definition is in the .service file. There's a handy 'systemctl cat' command which will dump the full definition to the console, with any overrides included.

Which is how you actually do these things, btw. You're not supposed to poke around in the files and form a "mental map". You ask the system what you want to know. If you want to know what sshd needs to start, you do `systemctl list-dependencies sshd`. If you want to know what needs sshd, then `systemctl list-dependencies --reverse sshd`. The info comes out clearly, and nicely formatted. You can even graphically plot the boot process if you want to.

> When you're the sysadmin writing/modifying a service, which is the only time you're touching unit files, the abstract distinction becomes moot - there is little point to installing a new unit without also enabling it.

There certainly is a point. Some services can't be started without configuring them and have no possible useful defaults. Eg, anything that needs an external database. Or stuff that needs security considerations. Or stuff where there's optional functionality in a package, such as that you get smartctl and smartd in the same package. You might want to use the commandline tool without setting up monitoring.

> If you do fark up, make a loop, and your system doesn't boot, you don't diagnose where that happened but just boot the independent old configuration without the loop.

Why would you want a system where you can even make a loop? It should be impossible for it to happen in the first place.

> do tell me how the example I posted could break

Upstream starts off not setting FOO. You set FOO=bar. Upstream then sets FOO=baz and FOO_OPT=qux. Now the calculated config sees FOO=bar and FOO_OPT=qux, which was nobody's intention and is likely inconsistent.

> There's a handy 'systemctl cat' command which will dump the full definition to the console, with any overrides included... Which is how you actually do these things, btw. You're not supposed to poke around in the files and form a "mental map". You ask the system what you want to know.

I explicitly acknowledged this as part of my argument two comments back.

> Some services can't be started without configuring them and have no possible useful defaults

If a sysadmin is writing a unit file, it's because they're creating a new service to run. The generality of writing a unit file that an end user might want to run has dropped away.

> Why would you want a system where you can even make a loop? It should be impossible for it to happen in the first place.

Exactly! I'm talking about my own experience with systemd. I created a dependency loop with some ancillary high-level service, systemd responded by choosing an arbitrary place to break the cycle, and I was left with a completely unbootable system. (preemptively: I'm sure there is some bespoke command I could have run to shake that out before rebooting, but that flows into my larger argument)

> Upstream starts off not setting FOO. You set FOO=bar. Upstream then sets FOO=baz and FOO_OPT=qux. Now the calculated config sees FOO=bar and FOO_OPT=qux, which was nobody's intention and is likely inconsistent.

Possible issue, yes. If this is an intended software setting, then I'd expect the software to take such a possibility into account, and log something useful. If the admin is screwing around with something like LD_PRELOAD, that's on the admin. Still, you have a file with nothing but what the admin changes. Easy to notice and undo. You don't have to diff the file against the upstream.

> If a sysadmin is writing a unit file, it's because they're creating a new service to run.

Sysadmins don't write unit files most of the time, they're distro provided. Think here of something like gitea -- it can be provided by the distro, but it doesn't work unless configured first. Where are your git repos? Which database are you going to use? Where's the mail server? All of that needs to be set after installing the package and it makes no sense to try starting it until that's done. So the sane initial state for it is "installed, but not started on boot".

> Exactly! I'm talking about my own experience with systemd. I created a dependency loop with some ancillary high-level service, systemd responded by choosing an arbitrary place to break the cycle, and I was left with a completely unbootable system.

Well, that sounds not good, and I'd say there should be some sort of sanity check for that.

> I for one am very glad that systemd modernized things a bit and helped remove some of this stuff from the equation.

I, for one, I'm glad there are still options and that even Debian let the door open to, one day, switching to something else than systemd.

I'm running Devuan (Chimaera) now and it's fine (for my needs).

It's not all rosy: there's something to be said about the insane kitchen sink that systemd is. Having the PID1 so gigantic, with its tentacles everywhere in the system, ain't something the Linux community should be proud of. I'm very happy that, so far, Linus is resisting and nothing in the kernel is (yet?) tied to systemd.

Linux is about choices too. The day there ain't no choice anymore, I'm switching to a BSD.

I'm running Linus' Linux. Not IBM/Red Hat/Poettering's Linux.

PID 1 isn't particularly gigantic under systemd. Most actual work is delegated to other binaries. Eg, journald isn't PID 1.

Personally, I take the opposite stance. I have very little ideology besides things being open source. To me a machine is a tool to do a job, and if systemd does the job better, then that's the right tool, regardless of any violation of principles that it might involve. There's definitely good ideas in the Unix model, but if the model ceases to be useful, it's time to throw the bad parts out.

I also have no issues whatsoever with running Red Hat's Linux (I use Fedora). It does the job, it provides the tools I like, that's what matters.

It doesn’t matter if PID 1 isn’t gigantic if it depends on stuff that is.

There are legitimate concerns for PID 1 being small and stable, since if that doesn't work you have big issues.

The rest doesn't really matter. They're separate processes and many are task specific and not used if you don't need that particular bit of functionality.

Ah, yes. I was having a discussion with BSD folks about how I was missing the equivalent of "service reload sendmail", and they replied that I have no business managing my own mail server if I don't know I should send SIGHUP to the process. I said that's great but this is my home box and I don't see how it helps me to be an expert in every service that's running there. For large installations, sure, there should be a dedicated email person. But are smaller installations not allowed to run sendmail so that cron can send its messages to some central place?

> Ah yes, the good old days some people are mysteriously nostalgic about.

> who knows if all the services will come back after a reboot. Better not risk it.

This is not a story about how bad sysV was but about how people happily disregarded any sort of infrastructure rigor "in the good old days". Applying upgrades that would break their system then wrote workarounds on top of others for those who also didn't care to test the builds, was considered reasonable. This isn't considered a good plan today and it wasn't a good plan then.