Hacker News Reader

The main argument for using shared libraries isn’t memory or disk usage, but simply security.

If you have a thousand packages linking statically against zlib, you will have to update a thousand packages in case of a vulnerability.

With a shared zlib, you will have to update only one package.

If a vulnerability in a single library can cause security issues in more than one package, there are much more serious issues to consider with regards to that library than the need to recompile 1000 dependents. The monetary/energy/time savings of being able to update libraries without having to rebuild dependents are of far greater significance than the theoretical improvement in security.

The solution here is to build tooling to track dependencies in statically linked binaries. There is no inherent reason that has to be tightly coupled to the dynamic dispatch model of shared objects. (In other words, the current situation is not an inherent fact about packaging. Rather, it is path-dependent.)

For instance, many modern languages use techniques that are simply incompatible with dynamic dispatch. Some languages like Swift have focused on dynamic dispatch, but mostly because it was a fundamental requirement placed on their development teams by executives.

While there is a place for dynamic dispatch in software, there is also no inherent justification for dynamic dispatch boundaries to be exactly at organizational ones. (For example, there is no inherent justification for the dynamic dispatch boundary to be exactly at the places a binary calls into zlib.)

edit: I guess loading up a .so is more commonly called "dynamic binding". But it is fundamentally dynamic dispatch, ie figuring out what version of a function to call at runtime.

> The main argument for using shared libraries isn’t memory or disk usage, but simply security.

“The” main argument? In a world filled with diverse concerns, there isn’t just one argument that makes a decision. Additionally, security is one of those things where practically everything is a trade off. Eg: by having lots of things link against a single shared library, that library becomes a juicy target.

> With a shared zlib, you will have to update only one package.

We are back to efficiency :)

That's not a difference of security, only of download size (*). A statically-linking distro would queue up those thousand packages to be rebuilt and you would receive them via OS update, just as you would with a dynamically-linking distro. The difference is just in whether you have to download 1000 updated packages or 1.

And on a distribution like OpenSUSE TW, the automation is set up such that those thousand packages do get rebuilt anyway, even though they're dynamically linked.

(*): ... and build time on the distro package builders, of course.

The build time is pretty significant actually. E.g. NixOS takes this to the extreme and always rebuilds packages if their dependencies change. It can take days to rebuild affected packages, and you typically can't easily update in the meantime. This is worse if the rebuild breaks something since someone will have to fix or revert that before it can succeed.

For non-security updates, they can do the rebuild in a branch (staging) so it's non-blocking, but you will feel the pain on a security update.

In practice, users will apply a config mitigation if available or "graft" executables against an updated lib using system.replaceDependencies, which basically search and replaced the built artifacts to use a different file path.

True. I'm involved in Alpine, postmarketOS and OpenSUSE packaging and I've seen builders of the first two become noticeably slow on mass rebuilds. OpenSUSE tends to be fine, but it has a lot of builders so that's probably why.

It shall be noted that there is no need to recompile everything, when a library's patched: you only need to relink the executables and rebuild the packages

Still, this requires a somewhat bigger infrastructure

Security + build time isn't the issue, but security + dev/testing time is.

Maintaining a secure package of zlib takes linearly more time with more versions of it used.

All distros are manpower limited.

There's only one version of zlib, the distro-packaged version, in the scenario of this discussion.

As I read it, that's 100% wrong: Isn't the discussion precisely about using the distro-packaged version vs. having other versions statically linked?

> That's not a difference of security, only of download size (*).

It is a difference of security, because now when there's an issue with (e.g.) OpenSSL, instead of just the OpenSSL distro package team having to worry about it, the MySQL distro package team has to, the Postgres distro package team, Nginx, Apache, cURL, OpenSSH, etc.

All of those teams have to coordinate to release at the same time.

We're talking about distro packages, not upstream software releases.

> We're talking about distro packages, not upstream software releases.

Yes, by "the MySQL package team" I meant the MySQL distro package team. Edited to clarify.

But even if applied to the upstream team that releases from (e.g.) repo.mysql.org or repo.postgres.org it would apply: they're now worrying about all their dependencies as well their own code, instead of just specifying in the .deb or .rpm file "I need libssl3" and letting the OS take care of it.

It would become a huge duplication of effort to keep up on security updates.

At $JOB we have a general policy to always use the distro-supplied version of a package whenever possible, as if we compile it on our own and put it into /usr/local or /opt, we now have to babysit that code for security and such ourselves instead of 'outsourcing' it to the distro.

Ain't nobody got time for that.

It is a difference of security. As one example, Debian used to be more lenient on allowing packages to vendor and statically link libraries. Around 20 years ago, an important zlib vulnerability was found that could be exploited through malicious compressed data. Rather than just pushing out a fixed zlib package, the package maintainers had to scan the repos for all copies and versions of the zlib source, patch them individually, and test each package. To make matters worse, some packages had modified versions of zlib that needed custom variants of the patch. This is a ton of work to ask of a small group of volunteers, and leaves users vulnerable for longer.

Vendored copies of libraries is a special case that's unrelated to this discussion. This discussion is about a distro whose packages dynamically link to the distro's zlib package vs a distro whose packages statically link to the distro's zlib package.

I don't think vendored copies are a special case or unrelated. For example, Debian does not allow vendored copies of libs. You have to use the version provided by the system, even if it's out of date or not the one you prefer. This explicit disallowing of different code versions also greatly improves security and threat response.

This recently (ish) bit hyprland, because they used a pinned version of wlroots, and the version of wlroots on Debian wouldn't work. And they couldn't just dynamically link to this pinned wlroots because that's still not around.

If the version of wlroots in Debian, hypothetically, contains a vulnerability, then only one change is needed instead of two.

I would argue if you allow static linking you implicitly encourage vendoring. I can't imagine such a distro where static linking is allowed, but there are 0 packages which use different lib commits.

> This is a ton of work to ask of a small group of volunteerss and leaves users vulnerable for longer.

Perhaps consider purchasing a support contract if people's charity endangers your users.

> A statically-linking distro would queue up those thousand packages to be rebuilt and you would receive them via OS update, just as you would with a dynamically-linking distro. The difference is just in whether you have to download 1000 updated packages or 1.

That is a large difference

But more significantly, you're assuming the only software installed is from the distro. This isn't what OP is talking about (unless the distro screwed up their package dependencies to get the not found error discussed.

Outside the distro-provided software, you either have to rebuild them all manually or wait for the proprietary software to do it and ship you a fix.

my /usr is 24G (restricting just to binary dirs, seems 18G). If every single time glibc had an update (perhaps a handful of times a year), I had to update my entire /usr (or with a bit more intelligence, just the binary packages), I'd be burning at least 100G (if not much more) of SSD life a year.

in a world of SSDs that wear out, the concept of updating the entire install every single time, seems problematic.

Have you ever actually worn out an SSD by compiling too much? I ran gentoo on SSDs for several years, never had an issue. If most compilation is happening in memory, updating software from source shouldn't be significantly more damaging to drives than binary patching?

Does static compilation use the entire library instead of just the parts that are used? If I'm just using a single function from this library, why include everything?

That's also a good argument for shared libraries, but the memory usage argument is a big one. I have 583 processes running on my Linux box right now (I'm posting from FF running on GNOME on Pop!_OS), and it's nice that when they run the same code (like zlib but also lots of things that are bigger than zlib), they load the same shared library, which means they are using the same mapped file and so the physical memory is shared. It means that memory usage due to code scales with the amount of code (linear), not with the amount of processes times the amount of code (quadratic).

I think that having a sophisticated desktop OS where every process had distinct physical memory for duplicated uses of the same code would be problematic at scale, especially on systems with less RAM. At least that physical memory would still be disk-backed, but that only goes so far.

That said, the security argument is also a good argument!

Kernel samepage merging and zram/zswap are the answers to the memory usage issue. The only actual issues with static linking is that it's harder to ensure that bugs and security vulnerabilites get patched.

I disagree:

- KSM only works on pages that are identical; I am skeptical that it can actually identify 2 instances of a lib after it's been through an optimizing compiler+linker (mostly LTO)

- KSM has performance overhead

- zram/zswap only let you reduce the impact of running out of memory at the cost of performance; you're still swapping, no matter how much we improve the details, so it'll always be worse than outright deduplicating libraries

Fair points overall. I think KSM is still useful and worth the overhead for things like sprawling browser tabs and densely deployed microservices.

I disagree that swap is only (or even primarily) for situations where you're running out of memory, though. It is useful for reducing I/O contention and improving memory utilization by replacing underutilized pages with file cache when it makes sense to do so. See this for more details: https://chrisdown.name/2018/01/02/in-defence-of-swap.html

Microservices quite likely, but I'd expect browsers to get ~zero benefit from KSM; aren't browsers all thin-copy forked from a single process anyways?

And yes I sort of agree with swapping unused stuff so you have more room for cache, but that's only good for unused stuff; if you swap the programs you're actually running - even just zram swap - you're going to hurt performance.

Same page merging is not going to work with static linking. The same modules linked into two different programs will likely have zero identical pages.

Zram/zswap isnt nearly as good as the kernel knowing straight from the outset that all processes want exactly the same mappings from the same files.

If you're in an adversarial relationship with the OEM software developer there's not a whole lot the distro maintainers can do, probably time to find a fork/alternative. (Forks exist for both your examples).

I say this as a casual maintainer of several apps and I'm loathe to manually patch versus upstream any fix.

I'm not going to switch to a firefox fork over one line in the configure script invocation. Forks have their own problems with maintenance and security. It's not useful to boil it down to one "adversarial relationship" boolean.

> I also care about using software from distro repos vs Flatpak etc wherever possible, because software in the distro repos is maintained by someone whose values align with me and not the upstream software author.

The problem one usually finds with distro repo packages, is they are usually out of date compared to the upstream – especially if you are running a stable distro release as opposed to the latest bleeding edge. You can get in a situation where you are forced to upload your whole distro to some unstable version which may introduces lots of other issues, just because you need a newer version of some specific package. Upstream binary distributions, Flatpak/etc, generally don't have that issue.

> the firefox package from distro repos enables me to load my own extensions without Mozilla's gatekeeping, the Audacity package from distro repos did not have telemetry enabled that Audacity devs added to their own builds, etc

This is mainly a problem with "commercial open source", where an open source package is simultaneously a commercial product. "Community open source" – where the package is developed in people's spare time as a hobby, or even by commercial developers where the package is just some piece of platform infrastructure not a product in itself, is much less likely to have this kind of problem.

With Debian, one can apt-pin different releases on as well. So you can run testing for example but have oldstable, stable, unstable and experimental all pinned on.

That maximizes your chance of being able to satisfy a particular dependency like libflac.8.so. Sometimes that might not actually be practical to pull in or might involve massively changing a lot of your installed software to satisfy the dependencies, but often it can be a quick easy way to drop in more libraries.

Sometimes libraries don't have a version number on them, so it'll keep being libflac even across major versions. Thats prohibitive because ideally you want to install old version 8 alongside newer version 12. But generally Debian is pretty good about allowing multiple major versions of packages. Here for example is libflac12, on stable and unstable both. https://packages.debian.org/search?keywords=libflac12

> Feel free to statically link it

Actually, please don't. There's a nontrivial chance that you'll ship a copy with bugs that somebody wants to fix.

Instead, to get ALL the advantages of static linking with none of the downsides, what you should do is:

* dynamically link, shipping a copy of the library alongside the binary

This allows the user to find a newer build of the same-version library. It's also much less likely to be illegal (not that anybody cares about laws).

* use rpath to load the library

Remember: rpath is NOT a security hole in general. The only time that can happen is if it is both a privileged program (setuid, setcap, etc.) and it contains a non-owner-writable path, neither of which is likely to happen for the kind of software people complain about shipping.

* do not use dlopen, please

`dlopen` works well for plugins, but terribly for actually-needed dependencies.

Install Gentoo.