Hacker News Reader

Comment by admax88qqq | Hacker News Reader

Unfortunately this is exactly the type of stuff that makes supporting commercial apps on linux a nightmare. Weird crashes due to weird linking of system libraries.

Common distros are very adamant about dynamic linking everything in order to support the use case of "core library has vulnerability, upgrade it in place without rebuilding consuming apps." Along with a desire to avoid "dll hell" and force a single canonical version of every library systemwide. This leads to these sorts of issues.

Windows gets around it by letting applications put the DLLs they care about beside the executable, and having it check there first by default.

View on HN

This is largely solved with approaches such as Flatpak or Snap, but graphics drivers are still an issue - they're expected to be supplied by the distribution, and components of them end up in-process in the application even if the rest of the application's runtime is shipped with the application. If there's an incompatibility between the application runtime and assumptions made by that driver code (as there appears to be in this case - TF2 ships its own malloc() implementation, but the graphics driver code ends up using it inconsistently and so blows up) then you're going to have problems.

I don't think there's anything about Windows that would fundamentally change things here. Windows apps aren't shipping their own graphics drivers, even if they're bundling everything else.

> TF2 ships its own malloc() implementation, but the graphics driver code ends up using it inconsistently and so blows up) then you're going to have problems. I don't think there's anything about Windows that would fundamentally change things here.

Yes, there is: on Windows, due to the way DLL linking works there, the graphics driver wouldn't use the malloc() implementation from TF2. The flat linking namespace in which you can globally replace the memory allocator for every dynamic library does not exist on Windows; if the graphics driver is linked to the memory allocator from the C library, it will get the memory allocator from that C library, not from some other DLL in the same process.

That's not to say Windows is free of dynamic linking problems. While on Linux it's mostly only NSS and the graphics driver (and only when explicitly requested), on Windows it's common for unrelated third party software to inject DLLs and threads all over every process on the system. And it's not uncommon for these injected DLLs to do things like hooking into system DLLs (by overwriting the entry point of exported functions, or even internal functions), leading to hard-to-diagnose crashes when things are not like they expected.

Linux does not have a flat linking namespace (for example, see RTLD_NEXT in the dlsym man page). If it did, this wouldn't be a problem because everything would use the libc malloc or everything would use the tcmalloc implementation. It's just that glibc exports pretty much everything as a weak symbol, most programs and libraries that link against libc bind to whatever ld.so picks as the first in the search object search order, and messing with linker settings is the last thing most developers want to think about.

I just read TFA so I haven't dived into all the details but my wild speculation is that the graphics driver is doing an aligned new and an unaligned delete. The alignment parameters in the new and delete give them different symbol names but you're free to implement them by dispatching to the unaligned version, so this wouldn't necessarily cause a problem (even in C++17 mode) if the malloc implementation doesn't actually handle them differently. However if tcmalloc was built for C++14 then it would only have the unaligned operators, and therefore the aligned new would resolve to the glibc implementation and the unaligned delete would resolve to the tcmalloc implementation, or something like that.

Not only Windows, Aix is also COFF based and has a similar approach to dynamic libraries, including export definitions and import files.

Isn't this what protected linkage is for?

>I don't think there's anything about Windows that would fundamentally change things here. Windows apps aren't shipping their own graphics drivers, even if they're bundling everything else.

Over on Windows, GPU drivers are provided and distributed by the manufacturer, Microsoft themselves might also distribute them through Windows Update.

GPU manufacturers also work together with Microsoft and bigger game dev studios (read: studios with sufficient cash/influence) to make sure everything works well together. The drivers are also signed off by Microsoft, both figuratively and literally.

Linux has none of this. Drivers are provided primarily by volunteers (most of whom couldn't care less about proprietary code), packaged and distributed by each distro, and most game devs couldn't care less about issues concerning less than one percent of their customers.

Getting consistent quality out of Linux drivers is usually much easier than Windows ones exactly because they're not supplied by a random hardware maker but are almost all upstream in the kernel. GPU drivers used to be a nightmare for this exact reason and are now finally all becoming properly upstream. Meanwhile the horror stories about Windows GPU drivers are all too common. Having to "clean up" old drivers. Having to stay on older versions because the new ones have a relevant broken feature, etc.

Let's be real, the two contenders for GPU performance (which, gamers appreciate) are Nvidia, which is not in the kernel, and AMD, which has an in-kernel driver, plus their own proprietary driver, same as Nvidia. TF2 is old enough that its runnable even on old Intel hardware, but let's not pretend that "almost all [drivers are] upstream in the kernel", so long as Nvidia and AMD both have proprietary, out-of-kernel-tree drivers, which mean that the "windows GPU horror stories" are also Linux GPU driver horror stories. Having to "clean up" old drivers. Having to stay on older versions because the new ones have a relevant broken feature, or the best: staying on an older version because the newer one doesn't support your hardware.

I've spent way too long on Nvidia's legacy unix driver page[1], but only Nvidia cards have Cuda support, or the performance, so I'm stuck there.

[1] https://www.nvidia.com/en-us/drivers/unix/

The proprietary AMD driver (AMDGPU-PRO) is only needed (or even an improvement over the open source version) in a few niche use cases. The majority of users are better off using the open source driver.

> These days our packaged drivers are mostly intended for:

> - customers using slower moving enterprise/LTS distros which do not automatically pick up the latest graphics drivers - we offer them both open source and proprietary/workstation options

> - customers using workstation apps who need the extra performance/certification from a workstation-oriented driver (although Marek has done a lot of great work over the last year to improve Mesa performance on workstation apps)

> The third target audience is customers looking for ready-to-go OpenCL, either for use with the packaged open/closed drivers or with the upstream-based stack in a recent distro.

https://www.phoronix.com/forums/forum/linux-graphics-x-org-d...

GPU drivers are notorious for being out of tree and even there most GPU drivers over the years have been upstream. Even if you limit to just GPUs AMD are not the same as Nvidia. The in-kernel AMD driver is the primary one these days and even Nvidia has finally relented and is finally helping move at least the kernel driver upstream. That Nvidia's closed-source driver has been a mess for all these years is just a further example of how the Linux model is much superior to the Windows one.

Can Linux not trivially do the same thing as windows with LD_PRELOAD? If so why is this more of an issue on Linux than Windows? Is it really less a technical challenge and more just a matter of Linux getting less support from upstream developers?

LD_PRELOAD is too global to be useful, it's hard to scope it to one process (and not child processes). macOS is better in the sense that it clears DYLD_* variables when the dynamic linker has done its work and the process starts. (Although that can also be painful when you want to run a shell script and set DYLD_* outside)

You can compile binaries with additional relative library paths in to them that will take priority over /usr/lib64

How? Maybe this should be better documented & recommended. I suppose at some point you're just statically linking with more steps - though for a problem like this it might be worth it.

See the ELF rpath, which can be set by the linker. This can be modified after using patchelf.

https://sourceware.org/binutils/docs-2.40/ld/Options.html#in...

The other comments have already covered the how, but I'd like to add that the mechanism used extensively in Nix [1].

[1] https://nixos.org/

$ORIGIN has been pretty well-known and documented for a very long time

You can set it in the environment for a single process.

I was thinking/wondering this myself. Not to reinvent the wheel - more toss an idea around, but a 'venv for LD_PRELOAD' sounds like it'd deal with this pretty handily

Not... in a way I'd use as a distribution/release maintainer. Probably as an administrator [of my LAN]

Such things already exist. Eg. Appimage or even docker.

and even that has been managed to be split between snap appimage and flatpak :D

(sorry not meant to offend, long time linux day-to-day user here, but it was just ironic for me to point out fragmentation of fragmentation ^^)

Right, but I don't really want to get into a distribution model - the hack suits me fine :)

More an exercise in curiosity than anything

Flatpak (or Snap, ew) probably deals with it fine today, Steam's there

That's Nix with extra steps.

I specifically said I'm not really trying to solution this, lol. More toying with the LD_PRELOAD aspect than anything

Nix is neat, and I don't think I've used it enough to be too critical - but in some ways it feels like 'extra steps'

I wanted to make a 'reproducible' installation (ala kickstart, not strictly binary)... but it felt very much like distribution work; declaring dependencies and the like

Oh, nix is an extra mile. A lot could be improved, but that's what I'm using to deal with dependencies.

Gotcha, I don't feel so floundering now!

I plan to spend more time with it, I see a lot of merit

The amount of control is great, but the docs could use some work. For my simple goals (install Sway, Ansible, some other things) it was a broadsword when I need a butter knife

What sold me on nix is home-manager and flakes: I can easily bootstrap my environment anywhere nix is available.

There are tools which overwrite linked libraries, eg: chrpath.

It can be done by setting rpath to origin, even post compilation using the patchelf tool. Works great with C shared libraries. Perhaps ABI issues with C++ shared libs introduces other problems.

With the warning that rpath!=runpath, both are called rpath, and which you get depends on your linker and whether you also pass -Wl,--disable-new-dtags

Runpath is the default, and also the one that is non-transitive and overridden by environment variables.

Yes linux _can_, the machinery is there, but culturally the common distros do not. And the defaults do not. On windows I can literally drop a DLL next to an executable and it will pick it up. On linux I have to do a wrapper script to set LD_PRELOAD, or mutate the binary's rpath to get it to load.

It's not really a question of capability, but a question of culture and defaults that makes linux hard to support.

Debian for example goes through great pains (or used to at least) to unbundle shared libraries such as openssl from projects like chromium.

This sounds like it's an interaction with the GPU driver though, which could also happen on windows...

Aehm. That is what a lot of closed-source applications do on Linux. And Valve does that, too.

The open-source ones are maintained in the packing system and kept lean.

The funny thing is in on Fedora in 2023 I don't feel like I'm missing out on most software.

Completely off base. If you want to distribute your application to users yourself (instead of letting the distro take care of that), then distribute all dependencies together with it.

There are a few dependencies that can not be easily vendored (At least not recommended). Mesa is probably the biggest example (An this case was caused by a mesa dependency). You can vendor them technically or even static link them but then you might end up with limited hardware support. The only alternative is to setup a mini opengl distro.

Which libGL.so should I be distributing alongside my application?

> Unfortunately this is exactly the type of stuff that makes supporting commercial apps on linux a nightmare. Weird crashes due to weird linking of system libraries.

That is the true reason containers were born, isn't it? The kernel is perfect, the public interface of the kernel is perfect. Userspace is a mess. Let's fix it by adding a layer between and have a userspace per application.

Isn't this exactly the use case for which flatpaks are designed? Isn't Redhat/Fedora in the process of adopting them as the primary way to support third party/proprietary graphical apps like Steam? Doesn't the current Steam flatpak avoid this issue?

TLDR; isn't this already addressed?

It would appear so, I don't understand why this blog post is so popular.

Reply on news.ycombinator.com