The state of GNU/Linux and a case against shared object libraries
mitjafelicijan.com
mitjafelicijan.com
Your problem with "libFlac.8.so is missing" happens with using software not from the distro repos. Feel free to statically link it, or run it via AppImage or Flatpak or Podman or whatever you want that provides the environment it *was* compiled for. Whether the rest of the distro is dynamically linked or not makes no difference to your ability to run this software, so there's no reason to make the rest of the distro worse.
I personally do care about disk usage and memory usage. I also care about using software from distro repos vs Flatpak etc wherever possible, because software in the distro repos is maintained by someone whose values align with me and not the upstream software author. Eg the firefox package from distro repos enables me to load my own extensions without Mozilla's gatekeeping, the Audacity package from distro repos did not have telemetry enabled that Audacity devs added to their own builds, etc.
I say this as a casual maintainer of several apps and I'm loathe to manually patch versus upstream any fix.
If you have a thousand packages linking statically against zlib, you will have to update a thousand packages in case of a vulnerability.
With a shared zlib, you will have to update only one package.
For instance, many modern languages use techniques that are simply incompatible with dynamic dispatch. Some languages like Swift have focused on dynamic dispatch, but mostly because it was a fundamental requirement placed on their development teams by executives.
While there is a place for dynamic dispatch in software, there is also no inherent justification for dynamic dispatch boundaries to be exactly at organizational ones. (For example, there is no inherent justification for the dynamic dispatch boundary to be exactly at the places a binary calls into zlib.)
edit: I guess loading up a .so is more commonly called "dynamic binding". But it is fundamentally dynamic dispatch, ie figuring out what version of a function to call at runtime.
“The” main argument? In a world filled with diverse concerns, there isn’t just one argument that makes a decision. Additionally, security is one of those things where practically everything is a trade off. Eg: by having lots of things link against a single shared library, that library becomes a juicy target.
> With a shared zlib, you will have to update only one package.
We are back to efficiency :)
And on a distribution like OpenSUSE TW, the automation is set up such that those thousand packages do get rebuilt anyway, even though they're dynamically linked.
(*): ... and build time on the distro package builders, of course.
For non-security updates, they can do the rebuild in a branch (staging) so it's non-blocking, but you will feel the pain on a security update.
In practice, users will apply a config mitigation if available or "graft" executables against an updated lib using system.replaceDependencies, which basically search and replaced the built artifacts to use a different file path.
Still, this requires a somewhat bigger infrastructure
Maintaining a secure package of zlib takes linearly more time with more versions of it used.
All distros are manpower limited.
It is a difference of security, because now when there's an issue with (e.g.) OpenSSL, instead of just the OpenSSL distro package team having to worry about it, the MySQL distro package team has to, the Postgres distro package team, Nginx, Apache, cURL, OpenSSH, etc.
All of those teams have to coordinate to release at the same time.
Yes, by "the MySQL package team" I meant the MySQL distro package team. Edited to clarify.
But even if applied to the upstream team that releases from (e.g.) repo.mysql.org or repo.postgres.org it would apply: they're now worrying about all their dependencies as well their own code, instead of just specifying in the .deb or .rpm file "I need libssl3" and letting the OS take care of it.
It would become a huge duplication of effort to keep up on security updates.
At $JOB we have a general policy to always use the distro-supplied version of a package whenever possible, as if we compile it on our own and put it into /usr/local or /opt, we now have to babysit that code for security and such ourselves instead of 'outsourcing' it to the distro.
Ain't nobody got time for that.
This recently (ish) bit hyprland, because they used a pinned version of wlroots, and the version of wlroots on Debian wouldn't work. And they couldn't just dynamically link to this pinned wlroots because that's still not around.
If the version of wlroots in Debian, hypothetically, contains a vulnerability, then only one change is needed instead of two.
I would argue if you allow static linking you implicitly encourage vendoring. I can't imagine such a distro where static linking is allowed, but there are 0 packages which use different lib commits.
Perhaps consider purchasing a support contract if people's charity endangers your users.
That is a large difference
But more significantly, you're assuming the only software installed is from the distro. This isn't what OP is talking about (unless the distro screwed up their package dependencies to get the not found error discussed.
Outside the distro-provided software, you either have to rebuild them all manually or wait for the proprietary software to do it and ship you a fix.
in a world of SSDs that wear out, the concept of updating the entire install every single time, seems problematic.
I think that having a sophisticated desktop OS where every process had distinct physical memory for duplicated uses of the same code would be problematic at scale, especially on systems with less RAM. At least that physical memory would still be disk-backed, but that only goes so far.
That said, the security argument is also a good argument!
- KSM only works on pages that are identical; I am skeptical that it can actually identify 2 instances of a lib after it's been through an optimizing compiler+linker (mostly LTO)
- KSM has performance overhead
- zram/zswap only let you reduce the impact of running out of memory at the cost of performance; you're still swapping, no matter how much we improve the details, so it'll always be worse than outright deduplicating libraries
I disagree that swap is only (or even primarily) for situations where you're running out of memory, though. It is useful for reducing I/O contention and improving memory utilization by replacing underutilized pages with file cache when it makes sense to do so. See this for more details: https://chrisdown.name/2018/01/02/in-defence-of-swap.html
And yes I sort of agree with swapping unused stuff so you have more room for cache, but that's only good for unused stuff; if you swap the programs you're actually running - even just zram swap - you're going to hurt performance.
Zram/zswap isnt nearly as good as the kernel knowing straight from the outset that all processes want exactly the same mappings from the same files.
That maximizes your chance of being able to satisfy a particular dependency like libflac.8.so. Sometimes that might not actually be practical to pull in or might involve massively changing a lot of your installed software to satisfy the dependencies, but often it can be a quick easy way to drop in more libraries.
Sometimes libraries don't have a version number on them, so it'll keep being libflac even across major versions. Thats prohibitive because ideally you want to install old version 8 alongside newer version 12. But generally Debian is pretty good about allowing multiple major versions of packages. Here for example is libflac12, on stable and unstable both. https://packages.debian.org/search?keywords=libflac12
The problem one usually finds with distro repo packages, is they are usually out of date compared to the upstream – especially if you are running a stable distro release as opposed to the latest bleeding edge. You can get in a situation where you are forced to upload your whole distro to some unstable version which may introduces lots of other issues, just because you need a newer version of some specific package. Upstream binary distributions, Flatpak/etc, generally don't have that issue.
> the firefox package from distro repos enables me to load my own extensions without Mozilla's gatekeeping, the Audacity package from distro repos did not have telemetry enabled that Audacity devs added to their own builds, etc
This is mainly a problem with "commercial open source", where an open source package is simultaneously a commercial product. "Community open source" – where the package is developed in people's spare time as a hobby, or even by commercial developers where the package is just some piece of platform infrastructure not a product in itself, is much less likely to have this kind of problem.
Actually, please don't. There's a nontrivial chance that you'll ship a copy with bugs that somebody wants to fix.
Instead, to get ALL the advantages of static linking with none of the downsides, what you should do is:
* dynamically link, shipping a copy of the library alongside the binary
This allows the user to find a newer build of the same-version library. It's also much less likely to be illegal (not that anybody cares about laws).
* use rpath to load the library
Remember: rpath is NOT a security hole in general. The only time that can happen is if it is both a privileged program (setuid, setcap, etc.) and it contains a non-owner-writable path, neither of which is likely to happen for the kind of software people complain about shipping.
* do not use dlopen, please
`dlopen` works well for plugins, but terribly for actually-needed dependencies.
400 megabytes of memory usage is probably worth more than 400 megabytes of storage. It may not be a make-or break thing on it's own, but it's one of the reasons Linux can run on lower-end devices.
If you've got 10-15+ consumers of a shared library or want to do plugins/hot-code reloading and have a solid versioning story by all means vend a DSO. If you don't however I would strongly recommend trying to keep all dependencies static and letting LTO/LTCG do its thing.
Also, sibling comments argue that kernel samepage merging can help avoid the bloat of static linking. But here what you argue will make every copy of the shared libraries oh-so-slightly-different and therefore prevent KSM from working at all. Really, no one is thinking this through very well. Even distributions that do static linking in all but name (such as NixOS) do still technically use dynamic linking for the disk space and memory savings.
The fact of the matter is outside of GPU drivers and poorly designed GUI applications, that's an extreme statistical outlier.
Just watch how these become used in every other app over the next few years.
And yes, the fact that you won't use those apps does not make it easier for the rest of us.
For the record, the 1.1GB executable I'm thinking about is a popular _terminal-only_ simulation package. No GUI.
I did a little math using a common GTK application as an example: nautilus. Using `ldd` and several shell utilities, if you were to statically link it it clocks in at around 80MB in binary size, but that would be worst case without LTO.
FWIW I think graphics stacks are one of the few places where dynamic linking is still highly relevant.
Abstractions are a thing in computer science. Abstracting at the shared library layer makes as much sense as abstracting in the RPC layer (which your software is most likely going to be obligated to do) or abstracting at the ISA level (which your software IS obligated to do). Your software has as many chances to break from a library change as it does from a display driver change or from a screen resolution change or from a processor upgrade. Why the first would bloat the "testing matrix" but not the later is over me, and already shows a bias against dynamic linking: you assume library developers are incapable of keeping an ABI but that the CPU designers are. (Anecdotally, as a CPU designer, I would rather trust the library developers..)
Many modern software development paradigms are simply not compatible with ABIs or dynamic binding. Dynamic binding also likely means you're leaving a bunch of performance on the table, since inlining across libraries isn't an option.
You'd be surprised, specially when I'm thinking 30 year old software. Again, usually I can patch around it thanks to dynamic linking...
> Many modern software development paradigms are simply not compatible with ABIs or dynamic binding
This is nonsense.
> Dynamic binding also likely means you're leaving a bunch of performance on the table, since inlining across libraries isn't an option.
Again, why set the goalpost here and not say ISA level or any other abstraction layer? I could literally make the same argument to any of these levels (e.g. "you are leaving a bunch of performance on the table" by not specializing your ISA to your software). How much are you really leaving? And how much would you pay if you remove the abstraction? What are the actual pros/cons?
The answer to each of these questions is specific to the circumstances. You have to decide based on general principles (what do you value?), the specific facts, and ultimately judgment.
I think in some cases (e.g kernel or libc) using dynamic binding generally makes sense, but I happen to think forcing shared library use has many more costs than benefits.
You're absolutely right that everyone should ask these questions, though. I work at Oxide where we did ask these questions, and decided that to provide a high-quality cloud-like experience we need much tighter coupling between our components than is generally available to the public. So. for example, we don't use a BIOS or UEFI—we have our own firmware that is geared towards loading exactly the OS we ship.
> This is nonsense.
Monomorphization like in C++ or Rust doesn't work with dynamic binding. C macros and header-only libraries don't work with dynamic binding either.
I don't know which OSs do this, but I know hypervisors certainly do this across multiple VMs.
> KSM was originally developed for use with KVM (where it was known as Kernel Shared Memory), to fit more virtual machines into physical memory, by sharing the data common between them. But it can be useful to any application which generates many instances of the same data
Although...
> KSM only operates on those areas of address space which an application has advised to be likely candidates for merging, by using the madvise(2) system call
1. You need a much better newer systemd than your distro likely packages.
2. KSM dedupe scans aren't free. Your system can spend its time doing that scan or it can spend its doing work with duplicate pages. Only relatively idle or highly homogenous systems would be free of the penalty.
3. For applications, especially statically linked ones, the duplicated code is not super likely to fall along page boundaries, thus the actual detectable duplication will be relatively low.
That said, it's still great for densely deploying a high traffic microservice that's more memory than CPU bound.
It's unlikely for the OS to effectively deduplicate memory pages from statically linked libraries across different applications.
I guess much of this is why its hard to use shared libraries in the first place.
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsp...
It would be true that this save memory if applications did not increase their memory requirements over time, but the fast is that they do, and the rate at which they increase their memory use seems to be dictated not by how much memory they intrinsically need but how much is available to them.
There are notable exceptions. AI models, Image manipulation programs etc. do actually require enough memory to store the relevant data.
On the other hand I have used a machine where the volume control sitting in the system tray used almost 2% of the system RAM.
Static linking enables the cause of memory use to be more clearly identified, That enables people to see who is wasting resources. When people can see who is wasting resources, there is a higher incentive to not waste them.
If we pay money to someone to audit our books, we are more likely to achieve more within our budget.
I assume there's something optimising that away but I'm not well versed.
That's what has made me leave Linux for Mac OS, and I'm talking about things like the touchpad (libinput) causing physical pain and crashing every couple minutes while the "desktop" wants to appeal to a hypothetical casual tablet user even Microsoft left long behind in the Windows 8.1 era (!). Mind, Mac OS is far from perfect and also regressing (hello window focus management, SIP, refactored-into-uselessness expose, etc, etc) but Mac OS has at least a wealth of new desktop apps created in this millenium to make up for it, unlike the Linux desktop struggling to keep the same old apps running as it's on a refactoring spree to fix self-inflicted problems (like glibc, ld.so) and then still not attracting new developers. I wish I could say, like the author, that containers are the solution, but the canonical example of browser updates also is a case of unwarranted and rampant complexity piling up without the slightest actual benefit for the user as the web is dying.
The alternative of supporting an insane amount of hardware, and making it work with every combination of software, both legacy and cutting-edge, is much, much harder to achieve. It's a miracle of engineering that Linux works as well as it does, while also being developed in a globally distributed way with no central company driving the effort. Microsoft can also pull this off arguably better, but with a completely different development and business model, and all the resources in the world, so it's hardly comparable.
The sad part is that there is really no alternative to Linux if you want full control over your devices, while also having a decent user experience. Windows and macOS are walled gardens, and on the opposite spectrum BSDs and other niche OSs are nowhere near as usable. So the best we can do is pick a good Linux distro, something we're thankfully spoiled for choice, and customize it to our needs, which unfortunately does take a lot of time and effort. I still prefer this over the alternatives, though.
Using GNOME is a personal choice that you made. There's lots of Linux distros that use better desktop environments like KDE and XFCE.
With AppKit, one can build just about any kind of app imaginable by importing a handful of stock frameworks. The need to use third party libraries is the lowest I've seen on any platform, with many apps not needing any at all. That's huge for lowering the amount of activation energy and friction involved in building things, as well as for reducing ongoing maintenance burden (no fear of the developer of libfoo suddenly disappearing, requiring you to chase down the next best fork or to fork it yourself).
The other thing is not being afraid of opinionated design choices, which is also a quality of AppKit. This is going to chafe some developers, but I see it as a necessary evil because it's what allows frameworks to have "happy paths" for any given task that are well-supported, thoroughly tested, and most importantly work as expected.
GTK and Qt are probably the closest to this ideal but aren't quite there.
- Qt Widgets has a selection of controls that falls a bit short of that of AppKit and doesn't receive much attention any more
- Qt Quick/QML seems more mobile-oriented and web-styled, being barebones if you don't pull in a library of third party widgets (e.g. Kirigami)
- Neither go as deep as AppKit with functionality and require more third party library usage
- Practically speaking, developers are limited to C++ or Python for Widgets and JavaScript for QML
- Distribution of Qt apps is messy and error-prone
These aren't really dealbreakers on their own but pile up to pull the overall experience down.
The problem with using game engines for UI is that their accessibility (support for screen readers, for example) is bad and they don’t adapt to the user’s desktop settings (e.g. light/dark mode and font size) unless you implement that yourself.
The downside however is that it's incredibly expensive compute and memory wise. Many applications are just too complex to represent as HTML + JS + CSS with a virtual dom in a performant manner. So then to make the performance work you have to write more custom code in JS or use more fringe technologies like WASM and at that point you're losing the usability aspect of Electron.
I haven't looked at the source code for VS Code, but I'm pretty confident in saying it's not naive/straight-forward HTML + JS.
I don't see this as a GNU/Linux problem, but a distro problem.
Regarding shared libraries: What do you do when a commonly used library has a security vulnerability? Force every single package that depends on it to be recompiled? Who's going to do that work? If I maintain a package and one of its dependencies is updated, I only need to check that the update is backward compatible and move on. You've now put a huge amount of work on my plate with static compilation.
Finally: What does shared libraries have to do with GNU/Linux? I don't think it's a fundamental part of either. If I make a distro tomorrow that is all statically compiled, no one will come after me and tell me not to refer to it as GNU/Linux. This is an orthogonal concern.
That's because while tracking dependencies is a solved problem, tracking whether each dependency correctly follows semver (especially in C land), and which packages and which "minor" patches break or don't break ABI, is not a solved problem.
Again... solved problem
They don't solve the dependency problem, they solve a distribution problem (in a bad for security way), what the AppImage provides is up to the author and once you go out of the "Debian/Ubuntu" sphere, you run into problems with distributions such as Arch and Fedora whom provide newer packages or do things slightly differently. You can have them fail to run if you're missing Qt, or your Qt version does not match the version it was compiled against, same GTK, Mesa, Curl etc.
The moment there's an ABI incompatibility with the host system (not that uncommon), it breaks down. Meanwhile, a Flatpak produced today should run in 20 years time as long as the kernel doesn't break user-space.
They don't run on my current distribution choice of NixOS. Meanwhile Flatpaks do.
In addition, the kernel breaks ABI all the time; sometimes this is partially workarounded thanks to dynamic linking (e.g. OSS and solutions like aoss). Other times not so much.
I feel that everytime someone introduces a "future proof" solution for 20 years they should make the effort to run 20 year old binaries on their Linux system of today and extrapolate from it.
* do not use static linking for libc, and probably not for other libraries either
C-level ABI is not the only interface between parts of a program. There are also other things like e.g. well-known filepaths, and the format of those can change. A statically-linked program has no way to work across a break, whereas a dynamically-linked one (many important libraries have not broken SOVERSION in all that time) will know how to deal with the actual modern system layout.
During my tests, all the statically-linked programs I tried from that era crashed immediately. All the dynamically-linked ones worked with no trouble.
The author seems to be focusing on the disk space advantage and claiming it's not enough to justify the downsides today. I can understand that but I don't think disk space savings are the main advantage of shared dependencies, rather it's centralized security updates. If every package bundles libfoo what happens there's a security vulnerability in libfoo?
That’s actually the key point that many people in this discussion seem to miss.
With multiple versions bundled to multiple apps, a good number of those apps will never be updated, at least not in a timely manner, and the computer will be left vulnerable.
But I feel pretty safe believing that the debian libfoo package maintainer is on top of things and will quickly release an update to libfoo.so that all apps running on my system will be able to take advantage of.
> Just boot yesterday's config and get on with your life
But then, that's one of those NixOS things that we take for granted.
But yeah, it’s pretty great to know that if your system fails, just `git restore --staged` and redeploy.
> The compiler does it well enough now that we don't have to
You know I see people say this and then I see some code with some nested loops running 2x as fast as code written with list comprehensions and I remember that it's actually.
"The compiler does it well enough now that we don't have to as long as you understand the way the compiler works at a low enough level that you don't use patterns that will trip it up and even then you should still be benchmarking your perf because black magic doesn't always work the way you think it works"
Struct packing too can still lead to speedups/space gains if you were previously badly aligned, which is absolutely something that can happen if you leave everything on auto.
The usual claim stands though, on a LoC basis a vanishingly small amount of code is perf sensitive (embedded likely more TBF)
On top of that, there is the all-packages.nix global namespace, which implicitly urges everyone to use the same dependency versions; but in practice just results in a mess of redundant names like package_version.1.12_x-feature-enabled...
The move toward flakes only replaces this problem with intentional fragmentation. Even so, flakes will probably end up being the best option if it ever gets coherent documentation.
This isn't really true. One version of nixpkgs (i.e. a specific commit of https://github.com/NixOS/nixpkgs) generally has one version of every package and other packages from the same nixpkgs version depending on it will use the same one as a dependency. Sometimes there are multiple versions (different major versions, different compile time options, etc.) but that is the same with other distros as well.
In that sense, NixOS is very similar to a more traditional distribution, just that NixOS' functional package management better encapsulates the process of making changes to its package repository compared to the ad-hoc nature of a mutable set of binary packages like traditional distros and makes it possible to see and rebuild the dependency graph at every point in time while a more traditional distro doesn't give you e.g. the option to pretend that it's 10 days or months ago.
You only really get multiple versions of the same packages if you start mixing different nixpkgs revisions, which is really only a good idea in edge cases. Old ones are also kept around for rollbacks, but those can be garbage collected.
There was a recent talk that discussed the security nightmare with dozens of different versions of shared libraries in NixOS and how difficult it is for the distribution maintainers to track and update them.
All of these new dependency bundling technologies were explicitly created to get out from under the abysmal state of packaging - from Docker (in some ways) and on to snap, flat pack, appimage, etc. This state of affairs was explained in no uncertain terms and widely repeated in the various manifestos associated with those projects. The same verbiage is probably still there if you go look. It seems crazy to act as if this recent memory is obscured by the mists of time, leaving us free to speculate on the direction of causality. We all lived this, and that’s not how it happened! Besides, in your telling, thousands of people and multiple separate organizations poured blood sweat and tears into these various bundling technologies for no good reason. Why’d they do that? I can’t help but suspect your answer is something like “it all worked fine, people just got lazy. Just simply work with the distro maintainer to get your package accepted and then … etc etc etc”. What do people have to do to communicate with the Linux people that this method of distribution is sucky and slow and excruciating? They’ve built gigantic standalone ecosystems to avoid doing it this way, yet the Linux people are still smugly telling themselves that people are just too stupid and lazy to do things The Right Way.
Because stability and rapid change are incompatible, and they wanted rapid change.
Which turns into a maintenance nightmare because now every app is using a different, incompatible version of the same library and somebody has to backport bug fixes and security updates to each individual version used by each individual package. And since that's a ton of work nobody wants to do, it usually doesn't get done and things packaged that way end up full of old bugs and security vulnerabilities.
My programs work, pass all tests on supported platforms, and don't have any active vulns. Forcing dynamic binding on me is probably not a good idea, and certainly not work I want to do.
(And I am skeptical of claims that leaf developers can keep up with the traffic of security updates)
For example, I'm the primary author and maintainer of cargo-nextest [1], which is a popular alternative test runner for Rust. Through its history it has had just one regression.
If I did ever release a new major version of nextest, I would definitely keep the old branch going for a while, and make noises about it going out of support within the next X months.
Security updates aren't that common, at least for Rust. I get maybe 5-6 alerts a year total, and maybe 1-2 that are actually relevant.
And in this wonderful world where developers are competent enough to manage this, and therefore there are no issues when libraries are updated (append-only, right?).... why do you have a problem with shared linking again? Or is this a case where you think yourself as an "above average" programmer?
There are also significant benefits to static linking (such as inlining and LTO) that are not relevant across process boundaries.
But yes, in a sense I'm pushing the problem up the stack a bit.
> is this a case where you think yourself as an "above average" programmer?
I've been very lucky in life to learn from some of the best minds in the industry.
That's great, but my confidence is very low that most maintainers are like you.
And note that static linking doesn't prevent third-party distributors like Linux maintainers from patching software. It's just that a lot of the current tooling can't cope too well with tracking statically linked dependencies. But that's just a technical problem.
GitHub's vulnerability tracking has been fantastic in this regard.
It's way too comfortable for _developers_ to bundle dependencies. That already explains why there is pressure to do so. You yourself look at this with developer glasses. I think users couldn't care less or may even actively avoid dependency bundling. Cause my impression, as a user, is that not only they almost never work right, but they actually make compatibility _harder_, not easier. And they decrease desktop environment integration, they increase overhead in every metric, they make patching things harder, etc. etc. Can you find other reasons why all these technologies you mention are not flying at all for desktop Linux users?
And speaking as a developer, the software I develop is usually packaged by distros (and not myself), so I'm very well aware of the "sweating" involved. And despite that, I will say: it is not as bad as the alternatives presented.
But the author already knows the solution...
> My best memories were always with Debian. Just pure Debian always proved to be the most stable system. I never had issue or system breaking after an update. I can't say the same for Fedora.
You've always had the power... tap your heels together three times, and just install Debian Stable.
In the case of firefox, it basically makes it less stable than a nightly build considering how often it crashes, and subtly breaks screen sharing in weird non-obvious ways. This experience has me guessing that Canonical probably cares more about server Ubuntu now than it does about desktop Ubuntu, which is a real shame.
While there are workarounds, I specifically endorsed Ubuntu in the first place to many people because these kinds of workarounds used to not be necessary. It's a real bummer honestly, not sure what else to recommend in this category either.
As for the Nvidia issues, especially the "system refuses to boot" kind, that's on Nvidia.
I haven't seen Fedora break in the last 2 years I've been using it, aside from a beta release upgrade I was curious to test that went wrong. I really think it's silly to put all the blame on shared libraries when there is a 99% chance it's the nvidia drivers fucking up again.
While I agree that disk usage is no longer a driver for shared libraries, memory usage still is, to some extent. If I have 50 processes using the same libaray (and I do), that shared library's readonly data sections get loaded into RAM exactly once. That's a good thing.
But even if that problem wasn't an issue, security is still a big one for me. When my distro releases a new version of a library package to fix a security issue, every single package that uses it gets the security fix, without its each maintainer having to rebuild each package against the fixed version. (Sure, some distros manage this more centrally and won't have to wait for individual maintainers, but not all are like that.)
I don't have to wonder what app has been fixed and what hasn't been. I don't have to make sure every single AppImage/Flatpak/Snap on my system that depends on that library (which I may not even know) gets updated, and possibly disable or uninstall those that haven't been, until they have.
I like shared libraries, even with the problems they sometimes (rarely!) cause.
But here's the part that feels wasteful that I wish could be solved. From version 1.0 to 2.0, probably 90% of most libraries are completely unchanged. Yet still we duplicate just to solve the problem.
What if instead of having a .so per version, we packaged together a computable shared object and had the compiler/linking system incorporate versions when making a link? From there, we could turn the requests for versions be something like "Hey, I need 1.2.3" and the linking system say "1.2.3 consists of these chunks from the shared repository". That could be manifest and cached by the OS.
For example, imagine fooLib has functions foo, bar, baz in version 1.2.3 and foo, bar, baz, blat. in 1.4.3. You could have a small hash of each of the functions in foo and store off the hash of those functions in a key value store for foo. From there you could materialize the runtime of foo 1.2.3 when requested by a library.
New versions would essentially be the process of sending down the new function chunks and there would be no overriding of versions. But, it would also give OS maintainers a route to say "Actually, anything that requests version 1.2.3 will get 1.2.3.1-patched because of a CVE". Or you could even hotpatch in the same function on all versions in the case of CVE having a more targeted patching system.
I've often wondered about if we could do a really granular dependency graph like this. Mainly because I like the idea of only shipping out the smaller changes and not 1gb of stuff because of what might break.
This hasn't been my experience at all, as someone who's been using it for more than 20 years now. Today, for the most part I don't have to tinker with anything, or think about the inner workings of my distro. Even just a decade ago that wasn't true.
apt-get install libflac7 libflac8
AppImages, do exist. What seems to be the missing "glue" is putting a ui in front of users to choose what software they want to run. That problem has not yet been solved gracefully.
The author says he knows how to use Linux, but observably never graduated to be a particularly advanced user.
It is not about graduating to being an advanced user. That's not the case at all. I know how to do it. I expect an advanced system to not force me to do it. That is my argument.
I don't think this will ever be solved. And that's ok, as well.
I think operating systems have been around long enough that this kind of things could have been resolved.
And Windows have the same issues. DLL's missing etc. This is not isolated to just Linux. And developers of applications have also a blame to carry here. I have done this as well.
edit: not even an advanced system, it's about how we package apps. That's probably a better way of saying it.
Strange reference because that video isn't relevant to the topic at all. It's about the difference between GNU coreutils command line options and other standards such as BSD and POSIX (the title is mostly a joke). In fact Luke Smith is vehemently against the idea of AppImages/Flatpaks/Snaps and believe they go against the spirit of GNU/Linux [1].
So now every app requires a different library version and you've achieved all the disadvantages of static linking combined with all the disadvantages of dynamic linking.
The modern reason is maintenance. If 100 apps are using libfoo, in practice nobody is going to maintain a hundred separate versions of libfoo. That means your choices are a) have hundreds of broken versions nobody is maintaining spread all over, or b) maintain a small number of major releases forked at the point where compatibility breaks, so that every version of 1.x.x is compatible with the latest version of 1.x.x and every version of 2.x.x is compatible with the latest version of 2.x.x, and somebody is maintaining a recent version of each series, so you can include it as a shared library that everything else links against.
But then you need libraries to limit compatibility-breaking changes to once or twice a decade.
Now it is not disk space, but for instance security that is a major theme. It is so nice to update the shared OpenSSL library and after that just being safe for every application linked to it.
However, it is indeed annoying that some distros (looking at you Arch) don't ship older versions of some shared libraries, like libLLVM. And then you indeed run into "shared library not found" issues.
WTF? On the contrary!
> And Snaps and Flatpaks tried to solve some of these things,
Made things worse.
> I don't care if these images are 10x bigger.
... the size is just part of the problem. The duplication is another part. A system depending on a 100K different versions of libraries/utils instead of, oh, say, 5K. And there's the memory usage, as others mentioned.
Also, there really isn't more trouble locating shared libraries today than 10 years ago; if anything, the opposite is true. Not to mention how there is even more searchable "crowd support" resources today than back then, for when you actually do have such issues.
So...
> How come GNU/Linux is worse than it was 10 years ago?
I think it's actually better overall:
* Less gotchas during installation
* Better apps for users' basic needs (e.g. LibreOffice)
* Less chance of newer hardware not being supported on Linux (it still happens though)
but if you asked me what is worse, then:
1. systemd.
2. Further deterioration of the GNOME UI. Although TBH a lot of that sucked 10 years ago as well (e.g. the file picker)
3. Containerization instead of developers/distributors having their act together
but certainly not what the author is trying to push. (shrug)
This is one area I think Linux is really bad on compared to Windows. AMD makes a new graphics card, it pushes out a driver update for Windows and it's all golden.
On Linux? They have to spend months getting it into the kernel release cycle, then wait on distributions to test that update, trickle it down to users and if you're on some kind of LTS distribution you might as well not bother.
You're not entirely wrong, either, though; it's more of a concern with LTS support. Though this is reduced somewhat by most things not actually needing custom drivers by using existing standards, e.g. HID, and/or by userspace drivers.
And, of course, you don't have it at all if you only buy hardware with Linux pre-installed and supported by the vendor. You know, like you do with Windows and Mac.
For all other scenarios, which are most, DKMS has existed for 21 years now.
The Linux distros' approach is superior to Microsoft's in every way. There will always be some Windows-minded types out there who will struggle, maybe they should buy a Mac.
DKMS is not a stable driver API/ABI.
I do know there are others out there, paid "professionals" staffed at some vendors, who struggle with DKMS. But not everyone paid to work with DKMS is necessarily competent by virtue of being paid to do a job.
I joined Linux late-game, and only recently discovered that AppImages were in some cases much nicer to work with. The thing that I was missing though was.. a package manager. If there was any distro that would build their package manager around AppImages, then I would gladly use it.
A couple hundred, not a couple...
Frankly, I don't think people are ambitious enough—whether it be the author of this post or those posting here in this thread.
We're at a good enough place with modern hardware—for personal/business computers, at least—that we should perhaps not even be thinking about object files at all. All software and all distros should focus on shipping software as source code; JIT everything at run time, but in such a way that JIT results aren't ephemeral—they get re-used, so by the second time (or nth time) it's the same as if all your utilities and system components had been AOT compiled. Neither the system operator, administrators, nor package maintainers should care any more about the object file format and the linking model than they care about... I dunno, the on-disk format of their browser cache for Web resources. (Quick, quick: do you know how your browser stores its cache items? Probably not, since it's not something anybody but browser developers think about.)
Fabrice Bellard already demonstrated with TCCBOOT the viability of applying this approach to the Linux kernel with acceptable results, and that was 20 years ago.
This would also help to ameliorate an unfortunate accident of history, where the fact that traditional software development and deployment has dealt in separate binaries and source code means there's friction for the user if they want to change how something works on their system. (This is true even if they've taken a hardline stance to ensure everything on their system is libre/open source—dealing with the build tooling on a project-to-project basis just isn't fun or tractable.)
This would also neutralize the entire schism between Flatpak vs AppImage vs Snaps vs <whatever>.
If relocation is required, then the DLL becomes a private copy in the process's address space rather than a memory-mapped file.
I believe this is one of the core issues here, and it is nicely illustrated by comparing what e.g. one Isaac Schlueter (isaacs of npm fame) thinks about dependencies and the critique to this offered by Rich Hickey (of Closure fame).
Basically what Isaac insists on is that Semantic Versioning can Save Us from dependency hell if we just apply it diligently. The advancement that npm offers in this regard is that different transitive dependencies that refer to different versions of the same module can co-exist in the dependency tree, which is great.
But sometimes, just sometimes folks, you need to have two versions of the same dependency for the same module, and this has taken a lot of effort to get into the system, because of the stubborn insistence that somehow `foo@4.1` and `foo@4.3` should be the 'same only different', and that really it makes no sense to use both `foo@3` and `foo@4` from the same piece of code, because they're just two versions of the 'same'.
Rich Hickey[1] cuts through this and asserts that, no, if there's a single bit of difference between foo version A and foo version B, then they're—different. In fact, both pieces of software can behave in arbitrarily different ways. In the real world, they most of the time don't, it's true, but also in the real world, the part of knowledge that I can be really sure of is that if foo version A is not bit-identical to foo version B, then those are different pieces of software, potentially (and likely) with different behaviors. Where those differences lay, and whether they will impact my particular use of that software remains largely a matter of conjecture.
Which brings me back to the OP's remark about libFlac.8.so conflicting with libFlac.12.so. I think they shouldn't conflict. I think we have to ween us off the somewhat magical thinking that we just need an agreement on what is a breaking change and what is a 'patch' and we can go on pretending that we can share libraries system-wide on a 'first-name basis' as it were, i.e. disregarding their version numbers.
I feel I do not understand Linux deeply enough but my suspicion has been for years now that we don't have to abolish shared libraries altogether if only we would stop to see anything in libFlac.8.so that ties it particularly closely to libFlac.12.so. Well there is, probably, a lot of commonalities between the two, but in principle there need not be any, and therefore the two libraries should be treated like any two wholly independent pieces of software.
[1] https://youtu.be/oyLBGkS5ICk?list=PLZdCLR02grLrEwKaZv-5QbUzK...
So yeah, these things sometimes happen.
Reads like a boomer rant from the greybeard who is resistant to adopting new technologies while complaining about how things were easier back in the day.
I have used Debanoids for decades and they just work. It's very nice.
Gentol works very nice too. I usually whack systemD and enjoy.
I do want to get a Fedora system going but eh. Just work is nice.
The BSDs are definitely not for everyone, and they come with their own set of tradeoffs. However, it is safe to say that all BSDs are better today than 10 years ago. Small and steady improvements over time.
Couldn't agree more.
It pushes for a single upstream for all of userland. With IBM being that single upstream.
After decades, I'll leave archlinux in my next migration because of this.
I hate to break it to you but what do you think the GNU Project is?