Hacker News Reader

Comment by duped | Hacker News Reader

It's really just glibc

It's really just not. GTK is on its fourth major version. Wayland broke backwards compatibility with tons of apps.

GTK update schedule is very slow, and you can run multiple major versions of GTK on the same computer, it's not the right argument. When people says GTK backwards compatibility is bad, they are referring in particular to its breaking changes between minor versions. It was common for themes and apps to break (or work differently) between minor versions of GTK+ 3, as deprecations were sometimes accompanied with the breaking of the deprecated code. (anyway, before Wayland support became important people stuck to GTK+ 2 which was simple, stable, and still supported at the time; and everyone had it installed on their computer alongside GTK+ 3).

Breaking between major versions is annoying (2 to 3, 3 to 4), but for the most part it's renaming work and some slight API modifications, reminiscent of the Python 2 to 3 switch, and it only happened twice since 2000.

The difference is that you can statically link GTK+, and it'll work. You can't statically link glibc, if you want to be able to resolve hostnames or users, because of NSS modules.

Static linking itself doesn't prevent modules. There's https://github.com/pikhq/musl-nscd for example

Not inherently, but static linking to glibc will not get you there without substantial additional effort, and static linking to a non-glibc C library will by default get you an absence of NSS.

You statically linked GTK+ won't work with the users UI styles though and probably won't support everyone's input methods or accessibility tools.

Multiple versions of GTK or QT can coexist on the same system. GTK2 is still packaged on most distros, I think for example GIMP only switched to GTK3 last year or so.

Can't we just freeze glibc, at least from an API version perspective?

Or just pre-install all the versions on each distro and pick the right one at load-time

We definitely can, because almost every other POSIX libc doesn’t have symbol versioning (or MSVC-style multi-version support). It’s not like the behavior of “open” changes radically all the time, and you need to know exactly what source symbol it linked against. It’s really just an artifact of decisions from decades ago, and the cure is way worse than the disease.

The problem is not the APIs, it's symbol versions. You will routinely get loader errors when running software compiled against a newer glibc than what a system provides, even if the caller does not use any "new" APIs.

glibc-based toolchains are ultimately missing a GLIBC_MIN_DEPLOYMENT_TARGET definition that gets passed to the linker so it knows which minimum version of glibc your software supports, similar to how Apple's toolchain lets you target older MacOS from a newer toolchain.

In principle you can patch your binary to accept the old local version, though I don't remember ever getting it to work right. Anyway here it is for the brave or foolhardy, here's the gist:

  patchelf --set-interpreter /lib/ld-linux-x86-64.so.2 "$APP"
  patchelf --set-rpath /lib "$APP"

Yes you can do this, thanks for mentioning I was interested and checked how you would go about it.

1. Delete the shared symbol versioning as per https://stackoverflow.com/a/73388939 (patchelf --clear-symbol-version exp mybinary)

2. Replace libc.so with a fake library that has the right version symbol with a version script e.g. version.map GLIBC_2.29 { global: *; };

With an empty fake_libc.c `gcc -shared -fPIC -Wl,--version-script=version.map,-soname,libc.so.6 -o libc.so.6 fake_libc.c`

3. Hope that you can still point the symbols back to the real libc (either by writing a giant pile of dlsym C code, or some other way, I'm unclear on this part)

Ideally glibc would stop checking the version if it's not actually marked as needed by any symbol, not sure why it doesn't (technically it's the same thing normally, so performance?).

Ah you can use https://github.com/NixOS/patchelf/pull/564

So you can do e.g. `patchelf --remove-needed-version libm.so.6 GLIBC_2.29 ./mybinary` instead of replacing glibc wholesale (step 2 and 3) and assuming all of used glibc by the executable is ABI compatible this will just work (it's worked for a small binary for me, YMMV).

> [...] brave or foolhardy, [...]

Heed the above warning as down this rpath madness surely lies!

Exhibit A: https://gitlab.com/RancidBacon/notes_public/-/blob/main/note...

Exhibit B: https://gitlab.com/RancidBacon/notes_public/-/blob/main/note...

Exhibit C: https://gitlab.com/RancidBacon/notes_public/-/blob/main/note...

Oh, sure, rpath/runpath shenanigans will work in some situations but then you'll be tempted to make such shenanigans work in all situations and then the madness will get you...

To save everyone a click here are the first two bullet points from Exhibit A:

* If an executable has `RPATH` (a.k.a. `DT_RPATH`) set but a shared library that is a (direct or indirect(?)) dependency of that executable has `RUNPATH` (a.k.a. `DT_RUNPATH`) set then the executable's `RPATH` is ignored!

* This means a shared library dependency can "force" loading of an incompatible [(for the executable)] dependency version in certain situations. [...]

Further nuances regarding LD_LIBRARY_PATH can be found in Exhibit B but I can feel the madness clawing at me again so will stop here. :)

Yes, so that's why freezing the glibc symbol versions would help. If everybody uses the same version, you cannot get conflicts (at least after it has rippled through and everybody is on the same version). The downside is that we can't add anything new to glibc, but I'd say given all the trouble it produces, that's worth accepting. We can still add bugfixes and security fixes to glibc, we just don't change the APIs of the symbols.

It should not be necessary to freeze it. glibc is already extremely backwards compatible. The problem is people distributing programs that request the newest version even though they do not really require it, and this then fails on systems having an older version. At least this is my understanding.

The actual practical problem is not glibc but the constant GUI / desktop API changes.

Making an executable “request” older symbol versions is incredibly painful in practice. Basically every substantial piece of binary software either compiles against an ancient Debian sysroot (that has to have workarounds for the ancient part) or somehow uses a separate glibc copy from the base system (Flatpak, etc.). The first greatly complicates building the software, the second is recreating Windows’ infamous DLL hell.

Both are way more annoying than anything the platforms without symbol versioning suffer from because of its lack. I’ve never encountered anyone who has packaged binaries for both Linux and Windows (or macOS, or the BSDs) that missed anything about Linux userspace ABIs when working with another platform.

Is it painful? Why? You need a build environment that has the old libraries. It does not have to be ancient, just exactly what you need.

It has to be as ancient as the oldest glibc you want to support, usually a Red Hat release with very old version and manual security backports. These can have nearly decade-old glibc versions, especially if you care about extended support contracts.

You generally have difficulty actually running contemporary build tools on such a thing, so the workaround is to use —-sysroot against what is basically a chroot of the old distro, as if cross-compiling. But there are still workarounds needed if the version is old enough. Chrome has a shorter support window than some Linux binaries, but you can see the gymnastics they do to create their sysroot in some Python scripts in the chromium repo.

On Windows, you install the latest SDK and pass a target version flag when setting up the compiler environment. That’s it. macOS is similar.

If the problem is getting the build tools to work in your old chroot, then the problem is still "people distributing programs that request the newest version", i.e. the build tool developers / packagers. I generally do not have this problem, but I am a C programmer building computational tools.

Look, it’s not that complicated. If you just build your software with gcc or whatever in a docker container with pinned versions, put the binary on your website, and call it a day, 5 minutes later someone is going to complain it doesn’t work on their 3 year old Linux Mint install. The balkanization of Linux is undeniable at this point. If you want to fix this problem without breaking anything else, you have to jump through hoops (and glibc is far from the only culprit).

You can see what the best-in-class hoop jumping looks like in a bunch of open source projects that do binary releases — it’s nontrivial. Or you can see all the machinations that Flatpak goes through to get userspace Mesa drivers etc. working on a different glibc version than the base system. On every other major OS, including other free software ones, this isn’t a problem. Like at all. Windows’ infamous MSVC versioning is even mostly a non-issue at this point, and all you had to do before was bundle the right version in your installer. I’ll take a single compiler flag over the Linux mess every day of the week.

Do you distribute a commercial product to a large Linux userbase, without refusing to support anything that isn’t Ubuntu LTS? I’m kind of doubting that, because me and everyone I know who didn’t go with a pure Electron app (which mostly solves this for you with their own build process complexity) has wasted a bunch of time on this issue. Even statically linking with musl has its futziness, and that’s literally impossible for many apps (e.g. anything that touches a GPU). The Linux ecosystem could make a few pretty minor attitude adjustments and improve things with almost no downside, but it won’t. So the year of the Linux desktop remains illusive.

> The balkanization of Linux is undeniable at this point.

Again this same old FUD.

The situation would be no different if there was only a single distro - you would still need to build against the oldest version of glibc (and other dependencies) you want to support.

The glibc has to be as ancient as the oldest one you want to support. The rest does not.

That's exactly what apgcc from Autopackage provided (20 years ago). https://github.com/DeaDBeeF-Player/apbuild

But compiling in a container is easier and also solves other problems.

The people will complain that glibc doesn't implement what they want.

The solution is simply to build against the oldest glibc version you want to support - we should focus on making that simpler, ideally just a compiler flag.

Reply on news.ycombinator.com