Hacker News Reader

I agree. I recall a talk from Linux Torvalds on how bad the glibc folks break things and why he won't ship a binary version of his scuba tool. If the binary breakages start with the darn C lib, you're gonna have recurring problems all the way up the stack on a regular basis I feel.

If the binary breakages start with the darn C lib, you're gonna have recurring problems all the way up the stack on a regular basis I feel.

In general glibc maintains an extremely stable ABI[1]. Forwards compatibility in both glibc and libstdc++ is something many companies depend on for their mission critical applications, and it's the entire reason Red Hat pays developers to maintain those projects.

[1]: https://abi-laboratory.pro/index.php?view=timeline&l=glibc

This has the case for 10 years with games on steam. The worst being libstdc++ ABI issues still around because many devs forget to statically link their c++ libs with -static-libstdc++. Because in windows, ABI stability is really good and then devs are used to that.

?? on windows you always have to ship the libc/libc++ along with your app, as part of the VS redistributables. That's pretty much the same than static linking, the symbols are just not in the same binary.

Not since Universal C Runtime was created, and made part of Windows 10 systems components.

As far as I understand, Windows solves this libc++ versioning issue using the side by side cache. I guess this is a little bit like flatpack.

Windows standard C and C++ ABI is stable since 2017. So the last 3 releases of the Visual C Compiler and C runtime hasn't changed the ABI.

However Windows also has a lower level system API / ABI i.e. Win32 that's always stable all the way to Win 95. WinSXS sometimes helps for certain legacy apps too. This allows apps using different C libraries to work together. Win32 contains everything about the OS interface. Creating windows, drawing things, basic text, allocating pages from the OS, querying users, getting info about files is all stable. Win32 is the lower level of the C library. There is also a better separation of the functions in different DLLs. Win32 has multiple DLLs that has different jobs and they are mostly orthogonal. Kernel32 contains core almost system call stuff. Shell32 contains interactions with the OS shell i.e. creating windows, message boxes, triggering different UIs.

The surrounding libraries like DirectX / DirectPlay / DirectWrite that are also used by the games are also part of the system libs starting from 7. They are also stable.

On Linux world there is no separation of the system level libraries and normal user libraries. Glibc is just another library that one depends on. It contains the entire lower level interface to kernel, all the network stuff and POSIX user access stuff. It also contains a component that has no job in a C library: the dynamic executable loader. Unlike Windows on the Unix systems the C library is at the lowest level and Glibc being the default libc for Linux and making itself the default provider of the dynamic executable loader makes writing stable ABI almost impossible. Since other libraries depend on Glibc and executables depend on Glibc's dynamic loader everything is affected by the domino effect.

I agree.

The elf loader should be extracted from the glibc... but there is price to pay, and it is where those guys could do their manic ABI breaking again.

Some very low level interfaces will have to be defined between the external elf loader and the posix/c runtime. For the moment those interfaces are private, just look at how intimate they are about threading and TLS.

Windows solves this issue by not making language runtime part of the OS libraries in a way that pollutes other libraries.

That means that you can have your application linked with library A version X, and load another library that is linked with library A version Y, and so long as certain practices are followed, everything works because you're not getting cross-contamination of symbols.

Meanwhile on Linux the defaults are quite different, and I can't load OpenGL ICD driver that depends on GLIBC_2.38 symbol on application that was loaded with glibc-2.37. Moreso, a lot of APIs will use malloc() and free() instead of language-independent allocator, unless the allocation comes from kernel. And no, you can't mix and match those.

This what the "pressure-vessel" container from collabora (used by valve on dota2/cs2), is trying to solve... but now the issue is the container itself as it does _NOT_ follow fully the elf loading rules for all elf binaries and does ignore many configuration parameters of many software packages (data files location, pertinent environment variables, etc), basically presuming "ubuntu" to be there.

Basically, I have my vulkan driver requiring fstat from 2.33 but the "pressure-vessel" (sniper version), has a glibc 2.31, then it does parse partially my global elf loading configuration and the configuration of some packages to import that driver and all its dependencies... including my glibc elf loader... ooof! That level of convolution will have a very high price all over the board.

I have arguments often with one of "pressure-vessel" devs because of some shortcuts they took which do break my distro (which is really basic and very vanilla, but not "ubuntu"). It took weeks to get the fixes in (I guess all of them are in... until the glibc devs manage to do something which will wreak havock on "pressure-vessel"). Oh... that makes me think I need to warn them about that super new and recent ELF relocation type they will have to parse and detect in host drivers...

On my side, I am investigating some excrutiatingly simple modern file format which should help a lot fighting those issues, there will be technical trade-offs obviously (but would run-ish on "old" kernels anyway with elf "capsules" and an orthogonal runtime). Like json is for xml, but for elf/pe.

It seems only the linux ABI can be trusted... til Linus T. is able to hold the line.

>It seems only the linux ABI can be trusted...

The ABI is not stable. Google has to do extra work monitoring for ABI breakages to make sure that pushing out an update of an LTS branch of the kernel does not break people's drivers.

https://source.android.com/docs/core/architecture/kernel/sta...

The module ABI is not stable, but that's not the ABI typically referred to when someone says "Linux ABI". That would be the userland ABI, which is stable.

Linux kernel userland ABI is stable. Nothing else is on linux.

What language independent allocator? I'm unaware of any memory interfaces in programming languages that aren't specific to that language.

What would you use instead of malloc and free?

The OS provided memory allocators - HeapAlloc, its wrappers GlobalAlloc and LocalAlloc (remainders of 16bit era), VirtualAlloc (page granularity, can be compared to MAP_ANONYMOUS with mmap(), and CoTaskMemAlloc which can be shared across COM processes + their respective "free" functions. malloc() is explicitly called out in documentation as "runtime dependant".

Similarly other OSes used to have memory allocation services that weren't linked in any way to a language runtime - VMS has several calls, all language independent, ranging from low-level equivalents of mmap() to malloc-alternative.

And yes, a big chunk of Windows portability is that various APIs use those language-independent calls internally, and so can developer in order to avoid creating issues - and documentation promotes those language independent methods.

At no point you get into situation with Windows API that you call free() on memory allocated by malloc() from a different libc.

In comparison, the available language-independent APIs without using any special libraries on Linux, are to directly call mmap() and sbrk() through inline assembly (beware the glibc wrappers!).

malloc in (g)libc isn't any more language dependent than HeapAlloc or VirtualAlloc. Both have a stable ABI that can be used by any language.

HeapAlloc and VirtualAlloc do not bring the whole language runtime with them, like glibc does, nor are they specified as part of a specific language runtime only (the unix primitive for allocating memory is sbrk() and mmap(), not malloc()).

And with how applications are linked by default on linux, whoever loads glibc first sets the stage for every other module in the program, which makes for "fun" errors when you get a library that happens to be compiled with newer version (or older). And even if you force windows-style linking (good luck, lots of infrastructure needed), you end up dealing with modules passing each other malloc()ed blocks and possibly trying to call free() from different allocator on it.

Anyway, for all practical purposes, glibc has no stable ABI at all.

I suppose they meant a "libc independent" allocator, e.g. jemalloc/tcmalloc/mimalloc/etc. Although, using such allocators comes with complications of their own: https://lwn.net/Articles/761502/

I think there are very few c++ ABI versions on windows and it is easy to select the one you want until it is installed, they can be there side by side.

If they're going to put the game on Steam then they should be using the Steam Runtime which is available here[1]. Otherwise they're shooting themselves in the foot.

[1] https://github.com/ValveSoftware/steam-runtime

> If it is already that hard for game binaries, good luck with apps.

Video games are some of the most complex and most difficult pieces of software both to build and to get right. Modern games are coded so poorly that driver makers have to release patches for specific games.

It's the gamedev world that can't get its shit together. Valve settled on an Arch-based distro for the Deck. The specific distro doesn't matter since Steam already establishes its system requirements and comes with a runtime.

Beyond that, I really don't see the issues you're talking about. Generally, any issues you have are fixed by symlinking the correct .so files. This is a side effect of targeting a specific version of software instead of keeping to its more base features. That's on the dev, not the user or distro.

You act like Windows has never had ABI breakage or versioning problems. I'd like to see the specific issues you ran into; maybe there's an easy fix like a symlink.

> Video games are some of the most complex and most difficult pieces of software both to build and to get right.

Actually as far as binary distribution goes, video games are some of the easier to build software as they tend to need limited operating system integration besides creating a (possibly fullscreen) window, getting user input, driving a graphics card to display the result and output audio somewhere. Only completely headless programs like command-line tools or servers have it easier. Contrary to the popular meme, creating Linux binaries for games which will run on all (current) distros and will keep running in the future is not rocket science. Not completely trivial, but something that any competent developer should be able to manage.

Where things get really hard is desktop applications which users will expect to have a much tighter integration with the rest of the environment. Neither Qt nor GTK have a long-term stable ABI and shipping them with your program just means you have a million more unstable ABIs you depend on.

> But you have engines really not ready, for instance electron based games: you don't have the right version of the GTK+ toolkit installed on your enlightenment/Qt/raw X11/wayland/etc distro? Nah, won't run.

Hm, not sure what you mean here.

At least with the nw.js toolkit (from which Electron was forked IIRC) I've never gotten a report of a distro where it would refuse to run because of an incompatible GTK version.

Why would a distro have GTK in the first place?

More reasonably, binary GUI apps should not expect more than the window system, on elf/linux, wayland with a legacy fallback to x11. It means, the GFX toolkit is an application choice on top of the windowing system. Binary GUI apps have to distribute it... they have to distribute their own version which would not conflict with the version installed on the user system... if any...

> Cuplrits are mostly glibc devs with their manic abuse of version names (and very recently, a GENIUS who added a new ELF relocation type): this is a pain for game developers to provide binaries which span a reasonable set of distros in time. Basic game devs install one of the latest mainstream and massive distros, build there, and throw the binaries on steam... but that recent distro had a glibc 2.36 and now their binaries have version names requiring at least a glibc 2.36 (often, rarely not). They have to force the right version names with... the binutils gas symver directive (see binutils manual) until all their version names are compatible with a glibc reasonably "old" (I would go for a reasonable 5 years... 7/8 years?):

Or ... just build against the oldest glibc they want to support.

Isn't the solution to this problem to just distribute the source code. Let package maintainers worry about creating binaries for their distros.

not when you actually want to sell the games for money.

Distributing source code is not an obstacle to selling games as for all but the most trivial games the meat is in the assets and scripts. You can open-source game (engines) while continuing to sell the rest - which is what e.g. allows you to run Doom on any device imaginable.

yes, but a game with non-free assets is still not going to be picked up by linux distributions to package, i think