Hacker News Reader

Comment by icedchai | Hacker News Reader

Call me skeptical. I haven’t seen a Linux system that put /usr on its own partition in at least 15 years.

I dont think that was quite the right argumentation. Or at least not clear enough.

In general, the desire is to put the system image into /usr. A single partition, that holds literally everything you need to launch an os. If you can mount a /usr, everything else will populate. If you want to atomically swap system images, just swap /usr.

Imagine the counter: what would happen if we just swapped /bin? Shit would be a mess. /lib and /usr/lib wouldnt get updated in sync. /include and /share wouldnt be in sync. The idea is that the Unified System Resources ought unifiedly be swapped out. And the system specific resources(/bin, /lib, /include, et ceteta) ought not exist, provide no real contemprary value, and making image swapping a much much harder to accomplish well.

I can buy that argument, but I’ve never seen /usr get swapped out with any modern package manager. How would that work? Or is this more theoretical?

FYI “usr” really means “user”. Somebody made up “unified system resources” as a backronym.

Yea, it’s pretty funny. It used to hold the home directories, but the disk holding the OS ran out of space, so they started putting new utilities in /usr/bin. The distribution tapes they sent out to everyone who ordered a copy were made directly from the live system, so we all inherited the same quirk. We’ve moved the home directories elsewhere, and now the whole OS is climbing into /usr in order to be atomically mountable. I keep thinking of that Gilbert and Sullivan song with the line “it gradually got so!”.

Even funnier, now a lot of software pollutes your home directory with shared files and dotfiles, so that space is invaded too. I resort to creating a separate files directory, either in /files, my home directory or in /mnt/files, and store all my local files there.

I’m not as bothered about that. I do have a git repository called dotfiles with some of the more important ones. I then use GNU Stow to install them from there to their customary locations. That helps me keep them in sync across computers.

Generally the idea is more for atomic system upgrades, and less for package managers. Why have package managers when you can just mmdebstrap a new image & try start it, but keep all your /etc and /home stuff? That's the dream. Package managers become semi-obsolete. I think there are a variety of ideas for how package managers might come into compliance & make use of this though! A lot of possibility. But the notion of OSes that can swap images, that transcend the idea of one special bucket of bits that we slowly move forward, is kind of new. Containers are 100% there, are making it clear. But the conventional OS is still lagging, learning, behind. I recommend checking out systems like Wyrcan to see a little deeper how we can eliminate some of the historical OS layers & build a more modular, more secure, more atomic, more application centric OS[1].

A lot of it relies on better filesystems like ZFS or Btrfs which have snapshots. I myself twiddled around with writing some Debian-Live boot tools that used btrfs snapshots, copied onto ram-drives[2], that would potentially work well with package managers (boot either your starting image, or your "new" snapshot image).

These ideas predate (the deeply reviled by some haters) Lennart's post, "Revisiting How We Put Together Linux Systems"[3]. But Lennart here, in mid 2014, understood what containers were doing, understood how atomic system upgrades were trending, understood what was happening. This isn't a Lennart idea, but Lennart as usual semi got the gist of what made sense, what would help, where we were going. He captured the good idea of what was occuring already into a post. And calling out /usr as "the system image" was a clear & obvious win for that day, back in 2014, and it's still a clear & obvious win today to consolidate the OS into one directory today. May it be so. This is the way. So say we all. It is known.

The "usr" meaning user is seemingly in fact true. But it's also, imho, wrong? The backronym is more truthful, better describes what role /usr plays & has played for multiple decades now. "user" doesn't really mean anything reasonable, clear, or cogent.

[1] https://gitlab.com/wyrcan/wyrcan

[2] https://github.com/rektide/debian-live-boot

[3] https://0pointer.net/blog/revisiting-how-we-put-together-lin...

You could achieve the same thing with sym links, which is basically what Apple does with newer versions of macOS. The main OS partition (/) is read only, essentially immutable (unless you disable SIP.) There is a writeable volume mounted on /private (in a funky way, using "firm links".) /etc, /var, /home, and a couple others are symlinks to /private/etc, /private/var, /private/home ...

They do some funky stuff with /usr/local and a few other directories, too. This person seems to explain it: http://www.swiftforensics.com/2019/10/macos-1015-volumes-fir...

It's the standard upgrade method on most deployed desktop/laptop linux, a certain Gentoo derivative known as ChromeOS, and from there it's also common update mechanism of Google COS (default for GKE nowadays) and Flatcar (not sure if Fedora mangling of CoreOS uses it anymore as they reportedly use rpm-ostree)

> standard upgrade method on most deployed desktop/laptop linux

Deployed desktop Linux has /bin and /lib, so it's somehow working.

ChromeOS has them as symlinks to /usr/bin and /usr/lib, so does Fedora and all Fedora-derived distros, so does Arch and its derivatives.

CoreOS Container Linux (haven't checked out Fedora CoreOS) and its' fork Flatcar Linux do that. I couldn't find any good doc about this, but the idea is that:

- RO /usr contains all the system, the rest is populated with tmpfilesd

- There is two /usr partitions, the active one is selected with GPT priority attribute

https://github.com/endocode/coreos-docs/blob/master/os/sdk-d...

It's another made-up problem whose purpose is to enable the mockery of a valid idea in Myth #11.

>In general, the desire is to put the system image into /usr. A single partition, that holds literally everything you need to launch an os

Hasn't this use case been replaced by initramfs?

Yeah, he (Poettering) intends /usr to contain everything needed to _run_ the OS, with the launch handled by initramfs.

Not really. A common spiel of the time was that having to mount / and /usr due to packager laziness started requiring it b handled in initramfs making it more complex. Previously / was supposed to be able to boot into minimal standalone system that could mount the rest of itself.

So the common argument was that one should accept that readonly separate /usr and snapshots and whatsoever you wanted weren't going to be a thing because fedora didn't want to maintain support for mounting /usr in initramfs.

Another idea: what if we don't ever swap images like this?

It looks like a bad idea that I would never want to do.

This is what your chroots and jails are for.

But if I wanted to do it in style, I'd hack up the cool kernel support to atomically mount multiple directories as part of one atomic swap.

How can you call it "atomic" if you have executables still running from the old /usr/bin? Say there is some script running that is executing external programs. At some unspecified point in its execution, /usr/bin/whatever changes meaning; all newly run programs are different versions on a different libc and so on.

If you have enough control of the system that there are no such processes, then you can easily do the swap directory by directory: just do it with system calls out of a single process, rather than by invoking tools, so that nothing trips up at the point in time when you have a mismatched /bin and /lib.

Nothing you'be talked about sounds like a remotely normal thing that is documented or discussed anywhere and I doubt there's more than a fistful of orgs on the planet that have jury rigged that up. Meanwhile there are at least a dozen distros doing atomic image upgrades. Fedora SilverBlue for one example.

These upgrades often involve a kexec restart. For fleets of servers- the target audience- it's expected that machines will go down & rolling restarts should be fine. That's how true atomic behavior is gotten. But I think you are nitpicking & being fussy with your complaining about old processes being left around; even if you do an in-place swap, but have some old processes around, at least the whole system is together & consistent. More broadly, thjs set of problem in general doesnt change much whether you are using a merged /usr or not.

I genuinely dont get any of your disgust at having a /usr image that contains the functioning system image. You talk about chroots, as though it's an alternative somehow, but this practice of having a self-contained image seems like it would greatly assist in quickly cloning new images to chroot into.

If "atomic" means "stop the world running one system, and reboot into a different system", we can easily arrange to reshuffle two or more directories. The program which does that can be easily designed not to be affected by the intermediate states, like where you have a new /lib, but still the old /bin.

Reply on news.ycombinator.com