Fact: This would make the separation between vendor-supplied OS resources and machine-specific even worse, thus making OS snapshots and network/container sharing of it much harder and non-atomic, and clutter the root file system with a multitude of new directories."
It should make a lot of tasks a lot easier. The one I like most is trivially setting up containers or VMs that have your base OS in them plus some new software that you are testing. For example, at work we use Debian with a bunch of extra packages installed, plus our own package containing all manner of software and configuration that we’ve written. Although we do a fair amount of unit testing, and the Rust compiler really has our back, it is very difficult to really test things outside of production. Currently we have to painstakingly set up a test environment on a production machine, using tricks like altering our PATH so that the test version of the tools we are working on will be called, or editing things so that they call the version under test instead. It’s doable but laborious.
If we were using Fedora instead of Debian, we would be able to use systemd-nspawn to spawn a container with the existing OS, read–only mounts containing our databases and other large datasets, and an overlayfs to redirect any writes to those file systems to an alternate location. Installing and running our software inside that container would work exactly as if we were in production, and testing everything would be so much simpler.
systemd-nspawn is available on Debian, but running it requires keeping extra copies of the OS for each container you want to spawn, so we haven’t actually gotten around to doing it yet. Oh, and each container needs a properly set up /etc directory. We can’t just leave it empty, and we can’t just blindly copy the host’s either.
A common example of "what's wrong with split /usr" that was pushed was things like dropping pci.ids database in /usr/share and it being required by code mounting /usr, and that mounting / and /usr separately in initrd was a wasteful complexity.
In fact, today is the first time I have heard of reasoning that supported /usr on separate partition and I thought it was just an inversion done later by people who noticed possible benefits for some extra initrd complexity (btw, dracut and systemd oriented initrd scripts are stupidly complex. Including generating systemd units by text templating in udev rules as part of dracut boot process. Wtf)
In general, the desire is to put the system image into /usr. A single partition, that holds literally everything you need to launch an os. If you can mount a /usr, everything else will populate. If you want to atomically swap system images, just swap /usr.
Imagine the counter: what would happen if we just swapped /bin? Shit would be a mess. /lib and /usr/lib wouldnt get updated in sync. /include and /share wouldnt be in sync. The idea is that the Unified System Resources ought unifiedly be swapped out. And the system specific resources(/bin, /lib, /include, et ceteta) ought not exist, provide no real contemprary value, and making image swapping a much much harder to accomplish well.
FYI “usr” really means “user”. Somebody made up “unified system resources” as a backronym.
A lot of it relies on better filesystems like ZFS or Btrfs which have snapshots. I myself twiddled around with writing some Debian-Live boot tools that used btrfs snapshots, copied onto ram-drives[2], that would potentially work well with package managers (boot either your starting image, or your "new" snapshot image).
These ideas predate (the deeply reviled by some haters) Lennart's post, "Revisiting How We Put Together Linux Systems"[3]. But Lennart here, in mid 2014, understood what containers were doing, understood how atomic system upgrades were trending, understood what was happening. This isn't a Lennart idea, but Lennart as usual semi got the gist of what made sense, what would help, where we were going. He captured the good idea of what was occuring already into a post. And calling out /usr as "the system image" was a clear & obvious win for that day, back in 2014, and it's still a clear & obvious win today to consolidate the OS into one directory today. May it be so. This is the way. So say we all. It is known.
The "usr" meaning user is seemingly in fact true. But it's also, imho, wrong? The backronym is more truthful, better describes what role /usr plays & has played for multiple decades now. "user" doesn't really mean anything reasonable, clear, or cogent.
[1] https://gitlab.com/wyrcan/wyrcan
[2] https://github.com/rektide/debian-live-boot
[3] https://0pointer.net/blog/revisiting-how-we-put-together-lin...
They do some funky stuff with /usr/local and a few other directories, too. This person seems to explain it: http://www.swiftforensics.com/2019/10/macos-1015-volumes-fir...
Deployed desktop Linux has /bin and /lib, so it's somehow working.
- RO /usr contains all the system, the rest is populated with tmpfilesd
- There is two /usr partitions, the active one is selected with GPT priority attribute
https://github.com/endocode/coreos-docs/blob/master/os/sdk-d...
Hasn't this use case been replaced by initramfs?
So the common argument was that one should accept that readonly separate /usr and snapshots and whatsoever you wanted weren't going to be a thing because fedora didn't want to maintain support for mounting /usr in initramfs.
It looks like a bad idea that I would never want to do.
This is what your chroots and jails are for.
But if I wanted to do it in style, I'd hack up the cool kernel support to atomically mount multiple directories as part of one atomic swap.
How can you call it "atomic" if you have executables still running from the old /usr/bin? Say there is some script running that is executing external programs. At some unspecified point in its execution, /usr/bin/whatever changes meaning; all newly run programs are different versions on a different libc and so on.
If you have enough control of the system that there are no such processes, then you can easily do the swap directory by directory: just do it with system calls out of a single process, rather than by invoking tools, so that nothing trips up at the point in time when you have a mismatched /bin and /lib.
These upgrades often involve a kexec restart. For fleets of servers- the target audience- it's expected that machines will go down & rolling restarts should be fine. That's how true atomic behavior is gotten. But I think you are nitpicking & being fussy with your complaining about old processes being left around; even if you do an in-place swap, but have some old processes around, at least the whole system is together & consistent. More broadly, thjs set of problem in general doesnt change much whether you are using a merged /usr or not.
I genuinely dont get any of your disgust at having a /usr image that contains the functioning system image. You talk about chroots, as though it's an alternative somehow, but this practice of having a self-contained image seems like it would greatly assist in quickly cloning new images to chroot into.