Testing my system code in /usr/ without modifying /usr/
0pointer.net
0pointer.net
I think this technique still remains common to escalate getting admin on a WordPress blog to taking over the host.
Laughed hard at the audacity of this one.
https://lwn.net/Articles/890219/ ("Debian still having trouble with merged /usr")
https://news.ycombinator.com/item?id=30929024 (288 comments)
Being "legacy" or not has nothing to do with the number of users. There are many cutting-edge distributions that (gasp!) do not even use systemd. Would you call those, "legacy"?
Look at the definition of the word, in the context of computing, according to the Oxford dictionary: "denoting or relating to software or hardware that has been superseded but is difficult to replace because of its wide use".
If anything, the most "legacy" distributions should be redhat and debian, and their respective derivatives.
PREFIX support is different for every second buildsystem upstream offers. You'd need to patch that, its a ton of work. Manpower that needs to be invested without having a commercially visible return - thats why commercial distros rarely do it and possibly never will if upstream does not start doing it out-of-the-box.
Rejecting the /usr split is worth it just for the purpose of not having to clean up after Lennarts ideas. He isn't the guy who carries the fallout, even when he is the one causing it.
>Rejecting the /usr split is worth it just for the purpose of not having to clean up after Lennarts ideas.
This makes zero sense, the main reason to do this is for compatibility with container tools and other things that benefit from having a simplified mount structure. It was not an idea that came from systemd and it also has nothing to do with "cleaning up after" anyone's ideas.
Have you ever written or maintained recipes for building software? PREFIX= support exists, but its not consistent and many packages need individual workarounds. Adjusting it is quite some work.
Which distro do you use? Made by whom? Be grateful that these guys do this work for you.
For obscure packages with a weird build system they often aren't doing the work for me, specifically because those build systems are hard to manage and nobody wants to deal with them. If you want distros to not package your software that's a great way to do it. If you want to discourage contributors from sending you patches because they can't figure out your build system then that's also a great way to do it.
If you're a big popular package like Chromium or Firefox then you can get away with having a weird build system but even then packagers don't want to deal with it past a certain point, hence browsers packaged in Snap and Flatpak...
Out of curiosity, what would be the ideal shipping method for you?
I agree that "giant makefiles" are awful, but a neat, simple and portable makefile is a beautiful sight to behold and a pleasure to run.
As someone who has had to hack those makefiles a lot, I can't really agree. There's no such thing as a "simple and portable makefile". The real way to generate "portable makefiles" is to use autotools. Every handwritten "portable makefile" I've seen actually has a ton of system-specific and compiler-specific rules. And when porting from Linux you're lucky if some package shipping a handwritten makefile doesn't have an undocumented dependency on GNU make.
Even something simple like PREFIX support has to be manually implemented and tested if you write your own makefile, and often the upstream project doesn't bother testing it and wants the distro to do it. If you want distros to package your stuff without any troubles and without making them a guinea pig for some weird ad-hoc build system then it would be easiest to just use one of the standard buildsystems, those are mostly just cmake, meson, autotools. A small cmake or meson script is significantly shorter than a makefile and is a lot more portable.
Ubuntu: 3 years ago
Debian: supported both merged and non-merged for a while; only merged supported since last year
Opensuse: last year (?)
Arch: btw
Fedora: 2011
...
Any big distro I am missing?
That said, there is a clear trend in distros to move to merge these directories and I don't see why not. There are some niche use cases for different partitions containing different binaries and such, but like all niche use cases you're going to need to configure those yourself.
The opposition's argument seems to revolve around "we always did it this way so we can't do it any other way", really. This is the stuff that led us to a world where the most popular distro now packages the standard browser in a Snap package.
> The opposition's argument seems to revolve around "we always did it this way so we can't do it any other way", really.
My understanding is that the technical objection is that Debian packaging tooling isn't up to snuff to deal with a merged /usr. There are various corner cases and technical debt that cause headaches.
And that the people promoting /usr are not doing the necessary work to get Debian to that point. That is they are not submitting useful patches.
So some patches and fixes have been submitted to correct the issue, but those have been blocked because they are not technically sufficient or high enough quality (??? not sure).
Something like that. I don't follow the drama. But that is my understanding last time I checked.
....
And then there is the issue of dealing with the conversion to merged /usr in existing systems.
For distributions like Fedora this is solved by putting the upgrade logic into the installer; anaconda. This way major changes, like init systems or merged /usr, is handled when you upgrade from one version to another.
Debian doesn't work that way. They want to be able to handle upgrades 100% through the package management system and debconf.
Debconf is essentially "install wizards" to allow people to choose configurations when packages are installed/upgraded and doesn't have equivalent functionality in the RPM world. One of the very few meaningful technical differences between deb and rpm.
Beyond all that there was past goes at attempting merged /usr, which depended on user selected preferences.
So this means that there are live systems that are a mixture of possible "/usr states". So those use cases need to be handled as well.
Which means that...
Yet again Debian politics has caused the creation of a vast amount of technical debt that must be overcome for it to move forward. Technical hurdles that are not experienced by other distributions. And because of that currently existing Debian installs can be in a sort of limbo half way between combined /usr and not, which is pretty much the worst possible outcome from a technical point of view. Ideally the change should be atomic in nature.
This is not terrible as many people place politics as a higher priority than technical excellence. So this state of affairs is actually desirable for many. Which is fine. Putting in extra work to help ensure that everybody is happy as possible with the outcome is not the worst possible thing you can do with your life.
My understanding is that the Debian people tried to get a merged /usr going without consulting the packaging people. The packaging people not only ran into technical issues, they also disagreed with the decision itself. Eventually some patches were made to make it sort of work, but also with a warning that merged /usr is basically unsupported and can lead to breakage (which is technically correct). Then someone else tried to remove the warning because it made the entire project look bad with all those warnings everywhere, and at this point it's just egos and politics from all sides of the argument.
I don't think there are that many actually maintained packages that will break in a merged environment. It's more about the theoretical issue and about principles than anything.
It started out as a technical issue (the invalidation of some core assumptions about the file system in critical packaging infrastructure), but it's so much more than that now. I think the conflict will blow over, but just like people are still advocating for the death of systemd it'll take many years.
I don't think this has caused as much tech debt as it all seems from the aggressive discussions. It's a problem with some packages and some very exotic system configurations (embedded stuff is great at creating those "technically correct and supported but we really shouldn't have encouraged this" edge cases) but for the vast majority of the users and maintainers the whole thing is in the past now.
In the Linux space there are so many things that actually matter, like the 5 second Firefox startup delay caused by Snap and the fact that GTK still lacks a proper thumb grid view after it's been a meme for so many years. This whole issue is kind of a useless thing to put so much energy into.
Qt has the same problem and, at this point, I doubt this will change at all in the next decade. Very few applications actually have any reason to use the default file chooser, and there are basically no developers who have any reason to modify it. If there were, they would have made the changes already sometime in the last "many years". The Snap issue is at least fixable with some caching.
If anyone's actually wondering: 2013.
IMHO systemd has amassed too many features, both documented and undocumented. Having worked with GNU+Linux for over 2 decades, I still find Devuan and Void easier to manage than Fedora and Ubuntu. The latter is fine for basic desktop needs though.
It seems like it is a solution for anything that lives in /usr and could break your system / cannot directly be put there because it's read only when you are developing it. Not systemd-specific though obviously the part that manages the system boot and system services is going to be particularly concerned by this.
> IMHO systemd has amassed too many features
But systemd is not some monolith that does a lot of things. It's a set of tools that do one thing, well, quite like GNU and that happen to be called systemd-something. Often, thin wrappers around kernel features, like nspawn. I find them very cleanly designed and easy to use. To each there own but I find the possibility to create services declaratively with systemd unit files particularly useful, instead of having to write custom shell scripts and copy paste boilerplate. Upstart was a step in the right direction in this respect. It's not just "fine for basic desktop needs", it's a great fit for desktops and also particularly helpful on servers as well (not speaking about Ubuntu, just systemd).
It. Does. Not. Matter.
Still to much code in one place. Not enough review and unrelated subproject bitrot at different rate without visibility.
https://projects.apache.org/projects.html
or GNU
https://directory.fsf.org/wiki/GNU
Or FreeBSD or OpenBSD....
Having a single large overall project that manages a diverse number of individual software products is a common pattern and certainly something that isn't remotely unique to Systemd. There is no reason to denigrate systemd for that.
Part of the reason why people seem to (falsely) think systemd is monolithic is because distros have been historically bad at packaging it and haven't split up the components, but that's been improving over time.
> Oh, and in case you wonder, all of this only works with distributions that have completed the /usr/ merge. On legacy distributions that didn't do that and still place parts of /usr/ all over the hierarchy the above won't work, since merging /usr/ trees via overlayfs is pretty pointess if the OS is not hermetic in /usr/.)
IOW, if I build a system with yocto for an embedded system, which is also a very useful candidate for a read-only system image installation, I cannot use this feature for this use-case because ... yeah because what. Because lennart says so. This got old so fast.
We can't blame the developers to not support any oddity and particularity that exists… and then blame them because the distribution makers adopt their tools because they work well (as a result?).
I'm happy they focus on a standardized base, this results in less development effort focused on more useful stuff, cleaner and more robust tools, it also makes it easy to switch between distributions and to target this standardized base when developing.
It also can be seen as a assumed limitation. Any tool has limitations. But the /usr merge has been conducted with reasons, and makes this kind of stuff actually possible. It's not like keeping a split /usr is very useful today anyway.
Systemd is not something distributions have to bend to. It's something they willingly adopted because it solves well problems they have.
And it's not just Lennart. Hate he gets gets old fast.
It's not an _assumed_ limitation. It's an _enforced_ limitation based on a limited world-view that also uses the gravity of the project to enforce the world-view on any user.
I'm not arguing pro or against split /usr. Reality though is that for various reasons there are legitimate use-cases for a non-unified directory tree, e.g. embedded ramdisks that contain only the barest of /bin in a crunchgen'd binary that don't want to duplicate stuff that's later used in /usr/bin. Linux is not only desktop and server.
systemd is being adopted because it solves problems well the distributors have, but THEN the distributor ALSO has to bend their distribution to the single enforced world-view the systemd folks have if they want it to integrate. They, the distributors, software integrators per se, are just forced to hack out limitations that are imposed by systemd folks for no benefit of either; systemd dev or user. No unified-/usr-user will benefit from this enforced restriction. A split-/usr-user will have to patch systemd and maintain their tree to use this feature if they have a good use-case for it because .. "systemd says so" (if you prefer that over the "B"DFL lennart).
Lastly, you are correct in that systemd is a pool of like-minded developers - not only lennart; and this is where I add: - who like to shape the reality out there instead of allowing distributors and users to easily take on their own responsibility easily. Sure, I can fork systemd (it's free software after all), but it's unnecessarily "opinionated" for its broad area of application. And that's where the "politic" dimension of systemd development comes into play ...
I don't think anybody is. It's their technical approach to develop stuff. But sure, their approach has a huge impact on the ecosystem.
> we don't allow to run it on legacy systems because it doesn't make sense
If I'm to design some piece of software that requires something, that this something is still relatively commonly not there and that I know it won't work without, yes, I'll go ahead, make a check and prevent the program to run in this configuration.
Think ./configure which will not write the Makefile if some critical dependency is missing. It's the same thing for me.
Anyway, the author did say "it won't work", he didn't say "We went out of our way to prevent the users from trying to do this if we detect an unmerged /usr". I think you read too much into the whole thing.
And this is the political dimension of systemd. One they keep using to shape the ecosystem to their preference.
> If I'm to design some piece of software that requires something,
See that's the point. It doesn't _require_ a merged /usr to work. The benefit of the merged /usr here is that this guarantees to potentially overwrite/extend "all" binaries, not "only" those in a split-/usr. The latter doesn't mean it doesn't work. It just means that they only overwrite one place, not all potential places something exists.
> And this is the political dimension of systemd. One they keep using to shape the ecosystem to their preference.
Any highly successful project is political then. Linux developers in particular force their kernel and their views on how a kernel should behave and be designed down the throats of their users.
... Obviously you have your say when people choose to use your stuff. But then that's fine, you are the expert in your domain. Who else is better placed than you? If people use your stuff, probably preferences are shared.
Or they are locked-in (through network effect for instance). But the systemd folks are not out there having an evil agenda they force the ecosystem into. They solve their problems and people do happen to adopt their stuff. They do seem to have a very "if it's broken, fix it" approach but that seems to work out quite well. Sure that's not always comfortable but I find it somewhat refreshing.
It is like saying that "It is great that engineers are working to provide a unified rail line across Europe, but they are assholes because they want everybody to use the same rail gauge as well as implementing these changes that will allow the rails to handle traffic going 200 mph. The unified rail folks have an agenda!!"
The problem they are trying to solve is solved by "bending of the will", so to say.
It isn't as innocent as you paint it.
systemd developers have other agendas when making decisions , you can hear it from horses mouth:
"Well, it is definitely our intention to gently push the distributions in the same direction so that they stop supporting deviating solutions for these things where there's really no point at all in doing so." [1]
And after watching systemd development for years I might argue they have a very liberal interpretation of what constitutes "senseless configuration differences between the distros for the really basic stuff"
[1] https://lists.freedesktop.org/archives/systemd-devel/2010-Se...
This is a major criticism against Linux distributions from the beginning. Especially from the *BSD camps.
And Systemd project is full of people actually putting the work into solving this major problem.
You are right that it isn't innocent. It is actually very sophisticated and probably very wise thing to do. It is helping a lot of people out.
The actual major problem is that systemd developers wasn't being contempt with just building an opinionated system and let it rise on its merits, instead they choose to use its status to "gently push" their opinions on everyone else.
Also you're ignoring the last part of that sentence: "where there's really no point at all in doing so". That's the most important part. If a distro has a real reason to deviate then that's fine. If a distro is deviating for no reason then that's doing a disservice to its users, and you can expect anyone who ships other things for that distro to be upset with them for pointlessly making extra work for everyone else. It's not just the systemd developers who will be inconvenienced.
I agree that this is precisely a nice use-case, but why is this a problem? I agree with Lennarts reasoning that it makes little sense if the files are arbitrarily split between / and /usr. In any case, Yocto has had support for usrmerge for a while now[1].
[1] https://docs.yoctoproject.org/ref-manual/features.html?highl...
No this is wrong. You can still use this feature but you get to manually type out all the mount paths yourself every time and work out all the bugs in it when you try to mount other things on top of it, because that's what you signed up for when you decided to have multiple paths. I'm really baffled as to how you could even remotely say this is a systemd problem. This will be a problem if you try to do any kind of bind mount or overlayfs stuff on that system.
I really suggest that you avoid making comments like "systemd uses its gravity to try and shape linux distributions and dictate how things 'ought' to work". That stuff is conspiracy theory territory at this point, it's so disconnected from reality that it's just pointless to say it.
More like systems with an immutable /usr have this problem, and systemd now has a way to circumvent those restrictions when developing. Fedora has been prototyping such systems for quite some time, not exactly with brilliant success so far, but it's good that they're trying.
How does systemd have a problem? Systemd doesn't care about having a read-only rootfs at all, except that it supports it and now ships a little tool that's useful if you happen to use them. Fedora and Ubuntu doesn't ship a read-only rootfs(1).
(1) Well, Fedora Silverblue is an experimental Fedora-variant that uses a read only rootfs. But that point still stands.
There are so many potential pitfalls of everything around init/initd, especially if downstream integration scripts have to keep up with upstream changes.
And I would argue that those should not be the burden of endusers, and also not be the burden of distro maintainers. Providing a settings file that can autolaunch your application in a failsafe manner for all distributions is where systemd shines at, like it or not.
While I agree that it isn't perfect and the debugging critiques are kind of valid, I also would argue that this can be solved with better (graphical) tools for the systemd ecosystem.
Also: if you assume that debugging a broken shell script (running as root) is easier than systemd, then your perspective is already biased, cause it's a skill set that requires lots of experience.
I mean, most shell scripts in /etc/init.d don't even set -e, let alone -u ... and should be considered a risk for system integrity.