Hacker News Reader

> If you find it hilarious it's because you don't understand the reason.

I think it's hilarious that you think users are supposed to understand "the reason" at this level of detail.

When we make a thing, we make choices about who it's for. Sometimes those choices are explicit. But this is a good example of an implicit choice. Clearly it grew into this shape for reasons, but those reasons are about the people doing the making, not the people using it.

Even if we grant that all of this hierarchy is necessary, it still imposes a large burden on the person trying to figure out "why is X doing Y?" And that might be fine if the makers said, "Well this is going to be hard, so what can we do to make it easy?"

But denying that users will reasonably find this hilarious (or frustrating or impossible) is just digging the hole deeper. Telling people that their understandable reactions are wrong doesn't change their reactions, it just gets them to stop mentioning the problems around you.

> it still imposes a large burden on the person trying to figure out "why is X doing Y?"

Systemd has tools for that, such as systemctl cat which shows all applicable configuration and what files they come from.

Personally I have found the systemd hierarchy to be very useful. It allows me to create overrides for certain properties in a Unit without having to worry about keeping the rest in sync with upstream, or my package manager overwriting my configuration. And it allows me to have different independent packages (or chef recipes or ansible playbooks) that create different override files for the same unit without stepping on each others toes.

Yes it is a little complicated, but for most end users, you don't really need to worry about most of the complexity, you just need to know where you put your overrides. And for distributors and power users, that complexity serves a useful purpose.

> Systemd has tools for that, such as systemctl cat which shows all applicable configuration and what files they come from.

Sure, it could be that systemd does have amazing tooling for helping users with the burdens they have created. Or, given the very mixed feelings about systemd, maybe they've tried to be user focused but not hit the mark. I don't know enough to say.

But my point is that either way responses of the form "it's so easy, you just have to remember [9 paragraphs of detail users don't care about]" are part of the problem, not the solution.

Systemd didn't create this problem. But it solves the issue of distributions shipping config files as part of packages and then on every package upgrade having to reconcile between the distribution's config and your modifications. Now with the drop in system you don't need to do that.

I am not an expert but I think in general systemd has a lot of complexity but it's to handle existing issues in a better way. Some of the older init systems might be simpler to describe or get started but lead to more confusing situations in the long run.

"Make it as easy as possible, but not simpler". I think this applies perfectly here. Systemd has very complicated requirements due to its very unique position in a Linux distribution. And as a (power) end-user of it, you basically need you basically needs to touch only /etc

> Systemd has very complicated requirements due to its very unique position in a Linux distribution.

But are all those requirements actually necessary for the job it's doing? Or have its developers imposed artificial constraints through their choices in designing systemd that make it more complex than it could otherwise be?

The combination of distributions existing and believing they “own” parts of the system, and users needing to override settings means it is as simple as possible, and no simpler.

Put more generically, I think we could add: “unless you build it all yourself, you’re going to have to work with other peoples decisions.”

A distribution 'owning' a directory means that the package manager is allowed to change files in there, and allowed to assume nothing major was changed from what it installed.

A distribution that doesn't own any directories like this effectively can't make any changes. It's like writing a class without private fields.

That's because some features are targeted at different audiences. End users want simplicity and ease if use. Admin also want those, but they are secondary to determinism, consistency, and ease of tracking changes.

I find the best way to think about systemd is as the result of some people that probably knew the most about init systems remaking init to be consistent, documented, and handle a bunch of new features that different newer init systems had used to good effect.

Then, because regardless of how much those people knew the vast mofority of information about needs was sequestered in 1000 different teams and projects, they had to add a bunch of new features. Repeat that 100 times, and you get what we have today, which is a system that mostly works (and works well if you actually adopt their preferred patterns), but one that also has the craziest and hardest to remember directive names I have ever seen, where slight nuance about how they function means entirely different params.

In a perfect world it would be rewritten with the same features but with all the knowledge of what features are actually needed known and consistently implemented with sane naming from the beginning, but nobody wants to go through that again.

That last bit is especially what interests me about systemd. From the way you describe it, it sounds like they made it mainly to please themselves, and were then forced to accommodate the actual users and actual users. But that they did so grudgingly and partially, again favoring their interests.

Assuming that's how the history went, fine. Everything has a history, and we have to deal with the now.

But my point is that we are not going to deal well with the now, let alone the future, unless we recognize the impact the history has on people. So if people defensively shout "that's not funny" and imply users are wrong for being bothered by the mess, then we're on track to build another mess. We can't do better until we are honest about why it didn't turn out as well as we would have liked.

> it sounds like they made it mainly to please themselves, and were then forced to accommodate the actual users and actual users

I think it's less that than that there were real advances they were after, which they delivered (Poetterings original essays on what it can and should do and where the ideas came from, such as OS X's LaunchD and some things from Windows, etc) referred to numerous things that every sysadmin I knew at the time that read it was excited if could be delivered (but none of use really appreciated what it meant to deliver them).

The issue is not that they initially built it to their own desired, but that the way SysV init worked with just launching bash scripts to start and control the process that eventually should be running meant that a lot of crazy behavior was put into scripts along the years, because when you can write a program to control your program, sometimes it's easier to just put special behavior into your launch script than to patch the real code in question in some way.

When this crashed against systemd's desire to know what's actually running and control the environment perfectly (which is beneficial, it was always a pain to have a process that started fine from your root user shell but not on startup), and strongly disincentivize programs between systemd and the process in question (which a bash script is) for reasons of process and resource tracking (Linux kernel cgroups were just starting to widely be used beneficially around this time), there started to be a mass influx of feature requests for behavior that used to be handled by dedicated bash scripts previously, but now systemd needed to account for.

Combine that with systemd adding real dependency tracking (as well as start order tracking, which is related but not quite the same, leading to the difference in Wants and Requires), and what we have is a system that's just trying to do something vastly more complicated than what SysV init did previously (and that's before we get to the scope creep which caused them to want to take over some other early boot processes to they could be deterministic or whatever, which is also a PITA for many people).

If you're actually interested in the history if this, I recommend reading "systemd, 10 years later: a historical and technical retrospective" at https://blog.darknedgy.net/technology/2020/05/02/0/. That's where most my info on the details come from. I was around in the beginning and was excited to see systemd start, but I didn't actually use it until far later since most the systems I administer don't get changes like that immediately (and it was in an extremely slow release cycle at that time).

If you read the above you may come away with a different perspective than me, but I will say that systemd is very consistent in its operation and very robust in how it tracks processes and whether there's been a problem and restarts are needed, etc, as long as you actually adopt their preferences (which is to say run everything foreground and let systemd deal with STDIN/STDOUT and environment, and have a good pid to start with). If you do that, writing a new service file is extremely easy. Far more easy than it ever was with SysV init. It's a different paradigm, and sometimes it can be a pain when some process doesn't want to play nice, but mostly it just works once you stop fighting it, which is nice, and cuts down on admin time figuring out problems (especially since it just grabs all STDOUT and STDERR for you so random error messages that don't go to logs aren't lost either).

Thanks for the details. I will check out the link.

I should say that "real advances they were after" is not at all incompatible with "made it mainly to please themselves". The things that pleased them may well have been "real advances"; they often are. The question is the extent to which they were prioritizing the needs, capabilities, and interests of the whole audience being served, not just the things they personally thought important or preferable.

Users are only supposed to know about `/etc`. The rest of the directories are useful for distribution maintainers, people who package software for a distribution (but may not be maintainers), and some developers.

> Distribution would like to override a default. It can patch the file but that is somewhat harder to discover (it would require reading the package source). Alternatively it can add a drop-in. That drop-in goes into /usr too since it's a distribution file.

> User would like to override a default. They should not edit /usr because that belongs to the distribution. They drop their override in /etc.

Larry Wall elegantly solved these problems 38 years ago with the "patch" utility. Debian is a good example of how its use has been standardized to solve these problems.

The patch approach to managing distribution-provided configuration files is superior to the systemd unit approach for at least three reasons:

- Non-experts can read the configuration files from top to bottom in linear order, so the documentation can be inlined by the distribution provider, which means they can add maintainers notes as necessary.

- It also works better than the systemd unit style of configuration for sophisticated users. In particular, if you add a now-incompatible option to the file in the suggested place (next to the documentation and commented out default version of the setting), then the package management system can detect that you've made a potentially breaking change before installing the new software, and ask you what to do. This works in the absence of sophisticated data modeling, schema mapping tools, etc.

- The implementation of all this is factored out and standardized across 1000's of packages, so it only needs to be implemented once and understood once.

Rebasing patches is pretty hard I'd say. Much harder than just explaining someone the logic behind /lib, /etc and /run. I mean, this directories are seperate for a reason. Knowing the idea behind the directory split (a.k.a. Linux Filesystem Hierarchy Standard) is more essential than knowing how to use the patch command line tool. There are more and more people who don't want to use the command line. And that's completely fine.

> - It also works better than the systemd unit style of configuration for sophisticated users. In particular, if you add a now-incompatible option to the file in the suggested place (next to the documentation and commented out default version of the setting), then the package management system can detect that you've made a potentially breaking change before installing the new software, and ask you what to do. This works in the absence of sophisticated data modeling, schema mapping tools, etc.

Anecdata, but still: Every single time the package manager asked me what to do, it was not because of a breaking change, but simply because the base file changed in a way that the patch didn't apply cleanly anymore. That is, I, the user, had to work around limitations of the patch format. If the distro configuration and the user configuration had been separate files, things would have just worked fine.

You don't have to be an expert to read multiple files. If you do need expert assistance with something like that please contact me and we can work something out.

The "expertise" is in finding those files and understanding their hierarchy

and the user doesn't touch /usr.

So it's not hilarious statements like this make it hard to understand the reason? I almost cannot tell whether your whole post is sarcasm, or just showcasing irony.

You mean that the top-level Linux directory are hilariously badly named?

If so, I don't think anybody will disagree.

But if you meant that's a large problem, then well, no it isn't. There are 7 of those badly named ones. It's a matter of reading half a page of text to see what they do, and after 2 or 3 times you read it as reference, you have them memorized.

If would be nice to solve the issue, but it isn't anybody's priority.

Or you'll never memorize them because those 2 or 3 times are many months apart since you're just a user that rarely tweaks those configs

But yeah, ease of use is nobody's priority

Yes, "The user (syadmin) doesn't touch /usr, except /usr/local", where compiled in applications reside with full path so /usr/local/{bin,sbin,share,...}.

If you're deploying an especially big application, you may also prefer /opt/$APPNAME/{bin,etc,usr,...} (A full filesystem tree, basically).

We manage a fleet of systems at work, and I don't remember editing or touching a file under /usr, unless the period I was developing .deb or .rpm packages to deploy on said fleet.

There are many written and unwritten rules in Linux world, and sometimes people disregard or ignore these rules so blatantly, so I can't understand whether the post I'm replying to is a sarcasm or irony.

The rules are very much written down. For instance, here are the rules for /opt:

https://access.redhat.com/documentation/en-us/red_hat_enterp...

There's also a section for /usr.

I’d start here:

• Filesystem Hierarchy Standard <https://www.pathname.com/fhs/>

• For Debian (and derivatives): <https://www.debian.org/doc/debian-policy/ch-opersys.html#fil...>

And, of course, hier(7):

<https://manpages.debian.org/stable/manpages/hier.7.en.html>

I think there is a gap here between those who routinely (exclusively?) used UNIX machines along with 50, 100, 1000 other people all logged in at once (generally pre-Linux era) and those who only ever got to see and enjoy a time when everyone could have a UNIX workstation on their desktop (Linux, Mac, or other x86 flavour, take your pick).

Yeah, I really feel like we're in a "horseless carriage" or "radio with pictures" era for computing. We have incredibly powerful machines available. But the dominant paradigm for managing that power is to slice it up into things that simulate much less powerful processors in software environments designed for a shared 1970s university department minicomputer.

In many ways I love it, of course. It's what I grew up on, and I'm sure I'll have a small Linux cluster in the retirement home, because that's how computers are "supposed to be". But I think the old paradigm has only lasted so long because we have so much computing power to burn.

I'm not sure I understand. When the "old paradigm" was new, computing power was much more limited. They couldn't afford to "burn" any computing power. But it's only stuck around, because now we have more computing power? That doesn't make sense

You might read some Kuhn. Paradigms change when the old one becomes unsupportable. But the vast increase in computing power per dollar means we can waste a lot of it without it being economically uncomfortable.

Well, most of us. A good check on this is a podcast where Dr Kevin Bowers is interviewed: https://www.youtube.com/watch?v=GlzWM5GKQT8

He works for a financial trading firm, where the competitive nature of it means that every processor cycle counts. For him, older paradigms are perplexing and irritating, because they are major barriers to getting the most out of the hardware.

Back in the shared era, the machines were better-suited to individual customization, and were more easily understood by system administrators. Now, they're optimized too... waste more computational resources when idle?

It's both hilarious and unintuitive, but for some people their personal familiarity trumps most other things

X11 clients and servers are now whispering "Hold my beer."

If you want to use /usr for yourself, be prepared to throw the FHS and compability with almost everything away and rebuild your whole userland with something like NixOS/Guix.

It's makes a lot more sense when you realize that /usr doesn't stand for "user", but "user software resources".

/usr stands for ‘user’.

In particular, in our own version of the system, there is a directory "/usr" which contains all user's directories, and which is stored on a relatively large, but slow moving head disk, while the othe files are on the fast but small fixed-head disk. — https://www.bell-labs.com/usr/dmr/www/notes.html

Isn’t that just a backronym? I read recently (here, I think) that on some original specific UNIX system, the user directories were there originally, and then other stuff that belonged elsewhere went in there because the other physical volume (where things like /bin were) ran out of space. Later another dish was added and profiles were moved there, to make /home.

/usr was added when Dennis Ritchie's and Ken Thompson's /home partition was full and they added another disk - or so the story goes.

unix system resources

unlikely semantic revisionism

https://unix.stackexchange.com/questions/103347/pronunciatio...

Nicely linked from one of the answers: https://youtu.be/tc4ROCJYbm0?t=809

Look, mac, I can use google unassisted.

Like I said, revisionism. Also known as a backronym. That's the entire joke.

I can't see how this doesn't make it more hilarious. It gives me "There's no fighting in the war room" vibes.

This is why /usr/local/bin is located in /etc right?

It’s still hilarious. The fact that there’s a need for this hilarity doesn’t negate that. What a mess humanity have created… (Let’s write a suckless OS!)

At least acknowledge how retarded this all sounds.