As to why they don't add an option to specify an older version? I don't know either and it is rather annoying to have to use docker images of older OSes to target older glibc versions. It's just one of many things that prevents linux from being as popular as windows for desktop users.
I don't think there's any technical reason why it couldn't be done.
To be fair to them though, Mac has the same problem. I worked at a company where we had to keep old Mac machines to produce compatible binaries, and Apple makes it hard to even download old versions of MacOS and Xcode.
I guess the difference is MacOS is easy to upgrade so you don't have to support versions from 13 years ago or whatever like you do with glibc.
You don't have to statically compile glibc, gcc just needs an option to tell the compiler to target say, version 2.14 instead of the latest one.
The newest glibc has all the older versions in it. That's why you can compile on say ubuntu 14 and have it run on ubuntu 24.
Many distributions do periodic mass rebuilds anyway and do not need that much long-term ABI compatibility. Binary compatibility seems mostly for people who compile their own software, but have not automated that and therefore couldn't keep up with updates if there wasn't ABI compatibility.
It's disproportionately annoying for open source projects who don't want to waste their time dealing with this.
Because glibc and ld/lld are badly designed. glibc is stuck in the 80s with awful and unnecessary automagic configure steps. ld/lld expect a full and complete shared library to exist when compiling even though it expects a different shared library to exist in the future.
Zig solves the glibc linking issue. You can trivially target any old version for any supported target platform. The only thing you actually need are headers and a thin, implementation free lib that contains stub functions. Unfortunately glibc is not architected to make this trivial. But this is just because glibc is stuck with decades of historic cruft, not because it's actually a hard problem.
Steps taken when? When building glibc? And - what steps?
> ld/lld expect a full and complete shared library to exist when compiling ...
But ld and lld are linkers...
> Zig solves the glibc linking issue.
But Zig is a language. Do you mean the Zig standard library? The Zig compiler?
> The only thing you actually need are headers and a thin, implementation free lib that contains stub functions.
Why do you need stub functions at all, if you're not actually using them?
Random blog post I just found about it: https://ruoyusun.com/2022/02/27/zig-cc.html
Not to say the ABI problem isn't real if you want to combine binary packages from different Linux-based OSes. Plenty of solutions for that have cropped up as well: containers, flatpacks, snaps, the list goes on.
Hard, hard disagree. The problems are somewhat comparable. But if any platform is more painful it's Linux. Although they're similar if you exclude glibc pain. At least in my personal experience of writing lots of code that needs to run on win/mac/linux/android.
> pretty much all developers simple resorted to shipping the OS system runtime with every package
Meanwhile Linux developers have resorted to shipping an entire OS via docker to run every program. Because managing Linux environment dependencies is so painful you have to package the whole system.
Needing to docker to simply launch a program is so embarrassing.
> except of course you can't even combine some modules compiled for debug with modules compiled without debug in the same executable
That's not any different on Linux. That has more to do with C++.
I have never needed docker "just to launch a program". Docker makes it easy to provide multiple containerised copies of an identical environment. Containers are a light alternative to VM images.
I assume you find the existence of Windows containers just as embarrassing? https://learn.microsoft.com/en-us/virtualization/windowscont...
Correct. The Linux architecture around a global of dependencies is, imho, bad and wrong. The thesis is it's good because you can deploy a security fix to libfoo.so just once for the whole system. However we now live in a world where you actually need to deploy the updated libfoo.so to all your various hierarchical Docker images. sad trombone
> Containers are a light alternative to VM images.
A light alternative to Docker is simply deploy your dependencies and not rely on a fragile, complicated global environment.
> I assume you find the existence of Windows containers just as embarrassing?
Yes.
I know my opinion is deeply unpopular. But I stand by it! Running a program should be as simple as downloading a zip, extracting, and running the executable. It's not hard!
In that case, I think you should stick to snap and flatpak.
I find it hard to discuss the merits of Linux va windows with regard to deploying software without addressing the elephant in the room which is the replacement of a collectively maintained system that ensure software cohabitation by the modern compartmentalised collection of independent programs.
Only if you choose to use docker. As other people have pointed out most things on Linux can be deployed using a package manager.
> A light alternative to Docker is simply deploy your dependencies and not rely on a fragile, complicated global environment.
So like flatpak?
> I know my opinion is deeply unpopular. But I stand by it! Running a program should be as simple as downloading a zip, extracting, and running the executable. It's not hard!
How is that better than apt install? That (or equivalent on other distros) is how I install everything other than custom code on servers (that I git pull or similar).
You can do that; just most choose not to and use packaging infrastructure instead. It's not mandatory.
Switching to static linking using a sane libc (not glibc) can be a pain initially but you end up with way less overhead IMO.
I think you need to quantify "very wasteful". Quite frankly it's actually just fine. Totally fine. Especially when the alternative has turned out to be massive Docker images! So the alternative isn't actually any better. Womp womp.
An actually elegant solution would be a copy-on-write filesystem that can deduplicate. It'd be the best of both worlds.
how does that work when you app needs access to the gpu drivers ?
> except of course you can't even combine some modules compiled for debug with modules compiled without debug in the same executable.
There's a good reason for this, which IMO Unix-like compilers and system libraries should start adopting, too. Debug and Release binaries like the standard C and C++ runtimes and libraries cannot be inter-mixed because they have have different ABIs. They have different ABIs because the former set of binaries have different type layouts, many of which come with debug-specific assertions and tests like bounds-checking, exception try-catch, null-dereference tests, etc.
> The Windows problem has always been so so much worse that pretty much all developers simple resorted to shipping the OS system runtime with every package and it's just expected nowadays.
This is not true at all. There are several layers to the counterargument.
Firstly, UCRT has supplanted MSVCRT since Visual Studio 2015, which is a decade old this year. Additionally, UCRT can be statically linked: use `/MT` instead of `/MD`. And linking back to the previous quote, to statically link a debug CRT, use `/MTd`. Set this up in MSBuild or CMake using release/debug build configurations. UCRT is available for install (and maintained with Windows Update) in older versions of Windows going back to Vista.
Next, Windows by default comes with several versions of C++ redistributables going back to Visual Studio 2005. All of these redistributables are also regularly maintained with Windows Update.
Finally, Windows SDK versions targeting various versions of Windows are available for all supported Visual Studio developer environments. The oldest currently available in Visual Studio 2022 is Windows XP SP3[1].
These all serve to thoroughly solve both combinations of backward-forward compatibility, where (a) the runtime environment is newer than the developer environment, and (b), the runtime environment is older than the developer environment.
It is perfectly possible to compile a single `.exe` on Visual Studio 2022 in Windows 11, and expect it to run on Windows XP SP3, and the vice versa: compile a single `.exe` on Visual Studio 6, and expect it to run on Windows 11. No dynamic libraries, no DLLs, nothing; just a naked `.exe`. Download from website, double-click to run. That's it. No git clone, no GNU Autotools, no configure, no make, no make install (and then f*cking with rpaths), nothing. Prioritising binary-only software distribution means Windows has prioritised the end-user experience.
> It's where the phrase "DLL hell" originated, after all.
This is also incorrect. 'DLL hell' is a pre-NT problem that originated when packagers decided it was a good idea to overwrite system binaries by installing their own versions into system directories[2]. Sure, the versioning problem was there too, but this is itself a result of the aforementioned DLL stomping.
[1]: https://learn.microsoft.com/en-us/cpp/build/configuring-prog... [2]: https://en.wikipedia.org/wiki/DLL_Hell#DLL_stomping
I fully daresay writing C and C++ for Windows is an easier matter than targeting any Unix-like. For context, see what video game developers and Valve have done to check off the 'works on Linux' checkbox: glibc updates are so ridiculously painful that Valve resorted to simply patching WINE and releasing it as Proton, and WINE is the ABI target for video games on Linux.
> There's a good reason for this, which IMO Unix-like compilers and system libraries should start adopting, too.
Why? I'm understanding from your comment that you can't mix debug and release objects on Windows because the ABI is different. That's not a feature, that's a limitation. If it works on Linux to mix debug-enabled objects with "release", what use would it have to make it not work anymore?
IIUC debug symbols can be totally separated from the object code, such that you can debug the release if you download the debug symbols. A well configured GDB on distros that offer this feature is able to do it automatically for you. It seems very useful and elegant. Why can't Windows do something like this and how is it an advantage?
(Genuine question, I have a remote idea on how ELF works (wrote a toy linker), not much how DWARF works, and not the slightest idea on how all this stuff works on Windows)
If windows is going to insist on different libraries in debug and release mode, I wish the development version of the library bundled debug and release builds together so I could just say “link with library X” and the compiler, linker and runtime would just figure it out. (Like framework bundles on the Mac). Windows could start by having a standard for library file naming - foo.obj/dll for release and foo-debug.obj/dll for debug builds or something. Then make the compiler smart enough to pick the right file automatically.
Seriously. It’s 2024. We know how to make good compiler tooling (look at go, Swift, rust, etc). There’s no sane reason that C++ has to be so unbelievably complex and horrible to work with.
Almost all do. Look for binary library releases; they almost always supply Debug and Release binaries together.
> Windows could start by having a standard for library file naming - foo.obj/dll for release and foo-debug.obj/dll for debug builds or something.
Funnily enough, it does. foo.lib for Release; food.lib for Debug.
Any bounds checks and assertions that rely on storing additional data such as valid iterator ranges or mutation counters would need to change the type layout, wouldn't they?
Even if the STL were purely a header-only library (and influenced only by code generation changes for debug builds), there's still the problem of ABI compatibility across different translation units--or different libraries--which might be built with different options.
EDIT: One of your sibling comments goes into greater detail!
But yes, C++ punts a lot of things to the build system, partly because the standard has to work on embedded systems where shared libraries don’t exist. A better build system could fix most of these things, but every build system that tries ends up massively complicated and slow, like Bazel.
I think browser vendors have the right idea when it comes to evolving standards. Vendors experiment using their own products and then come together and try and standardise their work at committee. I think that would be a much better idea than either doing nothing or, as you say, trying to boil the ocean.
C++ on Windows is perfectly OK, especially if you're building on Windows for Windows using MSVC or Clang-cl.
There is no difference between Linux and Windows here. The debug/release issue is ultimately up to the API developer.
C++ has has the standard template library (STL). libstdc++, libc++, and MSVC STL are three different implementations. STL defines various iterators. A common choice is for a release-mode iterator to be a raw pointer, just 8 bytes on 64-bit. But the debug-mode iterator is a struct with some extra information for runtime validation, so it's 24 bytes!
The end result is that if you pass an iterator to a function that iterator is effectively two completely different types with different memory layouts on debug and release. This is a common issue with C++. Less so with C. But it's not a platform choice per se.
> IUC debug symbols can be totally separated from the object code, such that you can debug the release if you download the debug symbols. A well configured GDB on distros that offer this feature is able to do it automatically for you. It seems very useful and elegant. Why can't Windows do something like this and how is it an advantage?
MSVC always generates separate .pdb files for debug symbols. Windows tooling has spectacular tooling support for symbol servers (download symbols) and source indexing (download source code). It's great.
Historically this might be because MSVC didn't even preserve the standard library ABI across versions (although it has done so for the last 10 years at least), so there were little ABI stability expectations.
"Linux" does not have a "compile my program in debug mode" magic toggle (or Release or whatever for what it's worth). Different IDEs and toolchains may have different defaults and expectations. "g++ -g" is not debug mode, it's debug symbols.
To generate debug symbols for a given binary (whether executable or library) on Windows and MSVC's cl.exe (and Clang on Windows), compile with `/DEBUG`[1] and one of `/Z7`, `/Zi`, or `/ZI`[2]. This is equivalent to `-g` on Linux gcc/clang. In particular, `/Z7` generates separate `.pdb` files, which contain debug symbols for the binary in question.
The options that the parent commenter and I were discussing, i.e. `/MD`, `/MDd`, /MT`, and `/MTd`[3] have to do with the C and C++ runtime link configuration. These correspond to multithreaded dynamic, multithreaded dynamic debug, multithreaded static, and multithreaded static debug respectively. Therefore, the small `d` refers to debug versions of the C and C++ runtimes. The differences between the debug and release versions of the C and C++ runtimes are listed in the following links[4][5][6][7][8]. The last link in particular demonstrates the debug CRT's functionality.
Conventionally on Windows, debug binaries are linked to the debug versions of the C and C++ runtimes; ergo the requirement that 'Release and Debug binaries on Windows cannot be combined'. This convention is respected by all maintainers who release binary libraries on Windows.
There is no equivalent on Unix-likes: it'd be like having 'debug' versions of libc.so.6/libstdc++.so/libc++.so/libpthread.so with different ABIs. If you wanted to change between release/debug here, you would have to at least re-link (if not re-compile) everything. Imagine having `-cstdlib=libc-debug` and `stdlib=libc++-debug` options.
Both sets of options (debug symbol options and C runtime link options) are orthogonal, and may be freely combined. Hence, it is perfectly possible to link the debug versions of the C and C++ runtimes to a 'release' executable, although it would be pretty weird. For instance, `/O2 /LTCG /arch:AVX2 /MTd`. Equivalent imaginary GNU-style command: `-O3 -flto=thin -march=x86-64-v3 -cstdlib=libc-debug stdlib=libc++-debug -static`. You can see what I mean, I hope.
[1]: https://learn.microsoft.com/en-gb/cpp/build/reference/debug-...
[2]: https://learn.microsoft.com/en-gb/cpp/build/reference/z7-zi-...
[3]: https://learn.microsoft.com/en-gb/cpp/build/reference/md-mt-...
[4]: https://learn.microsoft.com/en-gb/cpp/c-runtime-library/c-ru...
[5]: https://learn.microsoft.com/en-gb/cpp/c-runtime-library/crt-...
[6]: https://learn.microsoft.com/en-gb/cpp/c-runtime-library/debu...
[7]: https://learn.microsoft.com/en-gb/cpp/c-runtime-library/run-...
[8]: https://learn.microsoft.com/en-gb/cpp/c-runtime-library/crt-...
would be actually cool
it definitely does not. MSVC's debug mode is akin to for instance using libstdc++ with -D_GLIBCXX_DEBUG which does change the ABI. Just passing "-g" which enable debug symbols is very different from what Microsoft calls Debug mode, which adds very extensive checks at all levels of the standard library (for instance, iterators become fat objects which track provenance, algorithms check preconditions such as "the input data is sorted", etc.)
As for using an older version of glibc, _linking_ isn't the problem -- swapping out the header files would be. You can probably install an old version of the header files somewhere else and just -I that directory, but I've never tried. libstdc++ would probably be harder, if you're in C++ land.
This applies to other libraries as well because there are new(ish) math functions, strlcpy, posix_spawn extensions etc. that seem to be quite widely used already.
We used https://crosstool-ng.github.io/ to create such a "cross-compiler". Now it doesn't matter which distribution our developers use, the same source code will always turn into the same binary (reproducible builds, yay!) and those binaries will work on older distributions than the developers are using. This allows us to ship dynamically linked linux executables; our linux customers can just unzip + run, same as our windows customers, no need to mess with docker containers.
The downside is that we can't just use a library by `apt install`ing it, everything needs to be built with the cross compilation toolchain.
When linking with this library before glibc, the resulting binary will not depend on the new symbol versions. It will run on glibc 2.4 and on systems as old as Ubuntu 8.04 and CentOS 5 even when built on the most modern system.
Hence the reason why there is a separate API/ABI level embedded into a shared object file, which is pretty much always completely different from the library's package version because they broke compatibility so often.
Additionally there's lots of libraries that break their symbols and hash tables on every subminor update. (Usually when they use LLVM, but I'm not saying it's LLVM's fault)
For example, harfbuzz and libicu are a regular offender of this, meaning they fuck up every downstream project because the method signatures contain a randomized chunk which changes on every single subminor version.
If postgres prefers explicit library versions to prevent data corruption, then icu is doing things correctly and making sure that downstream consumers don't shoot themselves in the foot.
If you're using NULL terminated strings everywhere, and not UTF8/16, it's your own fault for doing so. Especially if you are building a database which should always contain a length of bytes or at least a CRC check before each entry's contents in its serialized file format to prevent exactly this from happening.
(Let alone the question why is there no schema version that should have prevented this from happening in the first place)
The dependency errors indicate real issues because of the way most distributions handle backwards compatibility: you have to build on the oldest version you want to support. Those errors happen if this rule is violated. For glibc-based systems, the effect is amplified because package managers have become quite good at modeling glibc run-time requirements in the package-level dependencies, and mismatches result in install-time dependency errors. Admittedly, I'm biased, but I strongly suspect that if we magically waved away the glibc dependency issues, most applications still wouldn't work because they depend on other distribution components, something that's just not visible today.
* Developers like to be on the latest-and-greatest distro, and rarely perform builds in a chroot. Sometimes this is sheer apathy; other times it is because backporting library dependencies is annoying.
* End-users can't even be assumed to be on the latest LTS.
Related, almost nobody understands `rpath` and why it should always be used instead of static linking (assuming normal dynamic linking doesn't work of course). There's some common FUD going around but unless you're a setuid or similar program it's not actually true (and even then, it's only an issue with some uses of rpath).