Fedora 38 LLVM vs. Team Fortress 2
airlied.blogspot.com
airlied.blogspot.com
Common distros are very adamant about dynamic linking everything in order to support the use case of "core library has vulnerability, upgrade it in place without rebuilding consuming apps." Along with a desire to avoid "dll hell" and force a single canonical version of every library systemwide. This leads to these sorts of issues.
Windows gets around it by letting applications put the DLLs they care about beside the executable, and having it check there first by default.
Not... in a way I'd use as a distribution/release maintainer. Probably as an administrator [of my LAN]
More an exercise in curiosity than anything
Flatpak (or Snap, ew) probably deals with it fine today, Steam's there
(sorry not meant to offend, long time linux day-to-day user here, but it was just ironic for me to point out fragmentation of fragmentation ^^)
Nix is neat, and I don't think I've used it enough to be too critical - but in some ways it feels like 'extra steps'
I wanted to make a 'reproducible' installation (ala kickstart, not strictly binary)... but it felt very much like distribution work; declaring dependencies and the like
I plan to spend more time with it, I see a lot of merit
The amount of control is great, but the docs could use some work. For my simple goals (install Sway, Ansible, some other things) it was a broadsword when I need a butter knife
Runpath is the default, and also the one that is non-transitive and overridden by environment variables.
It's not really a question of capability, but a question of culture and defaults that makes linux hard to support.
Debian for example goes through great pains (or used to at least) to unbundle shared libraries such as openssl from projects like chromium.
The open-source ones are maintained in the packing system and kept lean.
TLDR; isn't this already addressed?
I don't think there's anything about Windows that would fundamentally change things here. Windows apps aren't shipping their own graphics drivers, even if they're bundling everything else.
Yes, there is: on Windows, due to the way DLL linking works there, the graphics driver wouldn't use the malloc() implementation from TF2. The flat linking namespace in which you can globally replace the memory allocator for every dynamic library does not exist on Windows; if the graphics driver is linked to the memory allocator from the C library, it will get the memory allocator from that C library, not from some other DLL in the same process.
That's not to say Windows is free of dynamic linking problems. While on Linux it's mostly only NSS and the graphics driver (and only when explicitly requested), on Windows it's common for unrelated third party software to inject DLLs and threads all over every process on the system. And it's not uncommon for these injected DLLs to do things like hooking into system DLLs (by overwriting the entry point of exported functions, or even internal functions), leading to hard-to-diagnose crashes when things are not like they expected.
I just read TFA so I haven't dived into all the details but my wild speculation is that the graphics driver is doing an aligned new and an unaligned delete. The alignment parameters in the new and delete give them different symbol names but you're free to implement them by dispatching to the unaligned version, so this wouldn't necessarily cause a problem (even in C++17 mode) if the malloc implementation doesn't actually handle them differently. However if tcmalloc was built for C++14 then it would only have the unaligned operators, and therefore the aligned new would resolve to the glibc implementation and the unaligned delete would resolve to the tcmalloc implementation, or something like that.
Over on Windows, GPU drivers are provided and distributed by the manufacturer, Microsoft themselves might also distribute them through Windows Update.
GPU manufacturers also work together with Microsoft and bigger game dev studios (read: studios with sufficient cash/influence) to make sure everything works well together. The drivers are also signed off by Microsoft, both figuratively and literally.
Linux has none of this. Drivers are provided primarily by volunteers (most of whom couldn't care less about proprietary code), packaged and distributed by each distro, and most game devs couldn't care less about issues concerning less than one percent of their customers.
I've spent way too long on Nvidia's legacy unix driver page[1], but only Nvidia cards have Cuda support, or the performance, so I'm stuck there.
> These days our packaged drivers are mostly intended for:
> - customers using slower moving enterprise/LTS distros which do not automatically pick up the latest graphics drivers - we offer them both open source and proprietary/workstation options
> - customers using workstation apps who need the extra performance/certification from a workstation-oriented driver (although Marek has done a lot of great work over the last year to improve Mesa performance on workstation apps)
> The third target audience is customers looking for ready-to-go OpenCL, either for use with the packaged open/closed drivers or with the upstream-based stack in a recent distro.
https://www.phoronix.com/forums/forum/linux-graphics-x-org-d...
That is the true reason containers were born, isn't it? The kernel is perfect, the public interface of the kernel is perfect. Userspace is a mess. Let's fix it by adding a layer between and have a userspace per application.
The only way to make this break is if something is loading random unversioned solibs or whatever the latest one it can find is, and expecting this to work forever.
If it actually used a versioned solib, it would get llvm 15 just like it did before.
This is the whole point of versioned solibs.
The aligned allocation operators have existed since llvm 8.x.
The problem is not that the aligned allocation APIs are new. The problem is that TCMalloc is only partially replacing the global allocation APIs, it's just taken until this year for that bug to be exposed.
What has happened is presumably some part of the OS has updated its target C++ version so is now using the aligned allocators, which exposes the gap in TCMalloc.
I'm not sure if the spec explicitly allows an aligned allocation to be fed into an unaligned operator delete, but it seems like implementations do, so that's probably why adopting aligned operator new wouldn't be seen as an ABI break.
It's probably an allocation before the failing one that is being misued - so the backtrace pointing to openal doesn't necessarily mean it's openal's fault.
Running with valgrind or another heap memory checking tool will probably be helpful to track down that particular linked bug.
EDIT:
It looks like that there's at least one out-of-bounds write when starting up half life (On arch linux, so maybe slightly different library versions and not loaded the counterstrike mod).
It looks like a valve bug - writing 2 bytes at index [30] of a malloc'd size of 31 goes one byte over, and it looks from the backtrace it's all valve's code and not deep in some library that might have been loaded in. Writing 2 bytes to a string is a bit odd, perhaps it's trying to null-terminate but somehow uses a wstring null? Or some attempt at SIMD that isn't correctly bound?
It doesn't seem to crash for me though, it might just be luck that nothing important is put 1 byte over, and it feels a bit unlikely something would be due to allocation and type alignment requirements, but it's perfectly valid for the malloc implementation to keep something important in that byte.
Or perhaps there's some other dynamics that change this - it looks like it's doing stuff with paths, so may change size (of the allocation or even the amount written) based on where the steam app is installed - stuff like your user name length changing that may be the difference between a crash. Or even another issue somewhere else I didn't see, or valgrind didn't catch.
Just goes to show how many games ship for years with "big" bugs :P
For reference:
==27467== Invalid write of size 2
==27467== at 0x406526A: GetSteamContentPath()
(pathmatch.cpp:523)
==27467== by 0x4065927: pathmatch(char const*, char\*,
bool, char*, unsigned int) [clone .part.1] (pathmatch.cpp:594)
==27467== by 0x4066849: pathmatch (pathmatch.cpp:541)
==27467== by 0x4066849: CWrap (pathmatch.cpp:685)
==27467== by 0x4066849: __wrap___xstat (pathmatch.cpp:907)
==27467== by 0x406294A: stat (stat.h:455)
==27467== by 0x406294A: CFileSystem_Stdio::FS_stat(char const*, stat*) (FileSystem_Stdio.cpp:225)
==27467== by 0x4060819: CBaseFileSystem::AddPackFiles(char const*) (BaseFileSystem.cpp:1325)
==27467== by 0x4060AA4: CBaseFileSystem::AddSearchPathInternal(char const*, char const*, bool) (BaseFileSystem.cpp:254)
==27467== by 0x4060B37: CBaseFileSystem::AddSearchPath(char const*, char const*) (BaseFileSystem.cpp:186)
==27467== by 0x8049003: main (launcher.cpp:413)
==27467== Address 0x45e5f4e is 30 bytes inside a block of size 31 alloc'd
==27467== at 0x4041714: malloc (vg_replace_malloc.c:393)
==27467== by 0x4357C4A: strdup (strdup.c:42)
==27467== by 0x42F1A76: realpath_stk (canonicalize.c:410)
==27467== by 0x42F1A76: realpath@@GLIBC_2.3 (canonicalize.c:432)
==27467== by 0x406525B: GetSteamContentPath() (pathmatch.cpp:520)
==27467== by 0x4065927: pathmatch(char const*, char\*, bool, char*, unsigned int) [clone .part.1] (pathmatch.cpp:594)
==27467== by 0x4066849: pathmatch (pathmatch.cpp:541)
==27467== by 0x4066849: CWrap (pathmatch.cpp:685)
==27467== by 0x4066849: __wrap___xstat (pathmatch.cpp:907)
==27467== by 0x406294A: stat (stat.h:455)
==27467== by 0x406294A: CFileSystem_Stdio::FS_stat(char const*, stat*) (FileSystem_Stdio.cpp:225)
==27467== by 0x4060819: CBaseFileSystem::AddPackFiles(char const*) (BaseFileSystem.cpp:1325)
==27467== by 0x4060AA4: CBaseFileSystem::AddSearchPathInternal(char const*, char const*, bool) (BaseFileSystem.cpp:254)
==27467== by 0x4060B37: CBaseFileSystem::AddSearchPath(char const*, char const*) (BaseFileSystem.cpp:186)
==27467== by 0x8049003: main (launcher.cpp:413)Regarding the function, here it is: https://github.com/dreamstalker/rehlds/blob/master/rehlds/fi...
Interestingly, strdup gets compiled into:
89 04 24 mov [esp+101Ch+name], pszContentPath ; s
E8 82 DC 00 00 call strlen
66 C7 04 03 2F 00 mov word ptr [pszContentPath+eax], 2Fh ; '/'
*(_WORD *)&pPath[strlen(pPath)] = '/';`The compiler seems to completely inline the strcat and write the '/' and null as a single 2-byte word write, the null then being out of bounds of the malloc'd chunk and likely causing the error as it overwrites something important.
Interestingly, the open group spec says that a null argument to realpath is "Implementation defined" [0]
And the linux (glibc) man pages say it allocates a buffer "Up to PATH_MAX" [1]
I guess "strlen(path)" is "Up to PATH_MAX", but the man page seems unclear - you could read that as implying the buffer is always allocated to PATH_MAX size, but that's not what seems to be happening, just effectively calling strdup() [2]. I have no idea how to feed back to the linux man pages, but might be worth clarifying there.
[0] https://pubs.opengroup.org/onlinepubs/009696799/functions/re...
[1] https://linux.die.net/man/3/realpath
[2] https://github.com/bminor/glibc/blob/0b9d2d4a76508fdcbd9f421...
It would be a gross kludge, though.
a = ::operator new(some size, some alignment)
...
::operator delete(a);
The problem is that if operator new(size_t, align_val_t) is called then the struct has an alignment annotation. That can lead to codegen that reasonably assumes alignment, even without any source level decisions that depend on alignment. The result of having some equivalent of (either at runtime or link time)
void * operator new(size_t sz, align_val_t a) {
if (operator new(size_t) has been overridden) return ::operator new(sz);
...
}
The problem is that there is system code that they are calling that is making using of over aligned allocation, so therefore could be generating code dependent on said alignment. The failure mode can very easily be
someSystemLibrary.so`someFunction:
alignedThing = ::operator new(size, alignment)
...
i_dunno_dma_memcpy_or_something(a, somewhere else)
...
::operator delete(a)
But as the other comment mentioned - it should be a problem for tf2 in the first place since that's not the behaviour they're after.
I'm unsure what you're proposing here - the only methods you know in the replacement allocator are operator new(size_t) and operator delete(void). The two possible failure paths are:
a = ::operator new(some size)
...
::operator delete(a, alignment)
a = ::operator new(some size, some alignment)
...
::operator delete(a)
If you preload something that patches all the new/delete interfaces, you can do this without guesswork.
new(size, alignment) ->
res=alloc(size+alignment)
res_aligned=res+...
offsets[res_aligned] = res
new(size) -> alloc(size)
free(ptr) ->
free(offsets[ptr] || ptr)
offsets.del(ptr)You are absolutely correct that as a developer you can have your process override the allocator functions, and that is in fact what TF has done. The problem is that they have not overridden all of the allocation functions, and so they're crashing due to mismatching allocators being used. TF2 can "easily" fix this crash by implementing the aligned new, new[], delete, and delete[] operators in their custom allocator, or by simply removing their custom allocator's override of the global new & delete operators and using a common base class to get their faster allocator.
The question we're talking about is "how does the standard library respond to this scenario in a way that maximizes correctness?".
Now when a program does decide that they're going to override the global operator new and delete functions the standard library is required to default to them instead. So generally the standard library exposes them as weak symbols, and the OS and stdlib links to them by symbol name. That way on program launch the program's version of the operator new/delete symbols are what win. So that's how the OS and standard library are able to interact with the program despite it overriding what is ostensibly the system allocator.
So in principle the OS could simply make sure that the user provided operator new, delete, etc are always directed to the system allocator routines. The problem is that when compiling user code there's no obligation to call the user provided new, delete, etc through a symbol, and in general won't. Instead the calls will generally be compiled down to PC relative loads and branches as those are significantly faster. The net result is that while the OS _could_ force the symbols to always resolve to the system functions, things would break due to the user code still using the user specified functions, but those functions then would not be compatible and the result would be sadness. Hence the user defined operator implementations have to win.
The problem is what happens when not all of the operators are overloaded. This historically hasn't been a problem: there's the plain and [] variants, which can be overloaded independently, and the no_throw variants of each which have in practice not been an issue because the way those are implemented by default is essentially
try { return ::operator new(size); } catch(...) { return nullptr; }
The problem that operator new(size_t, align_val_t) is that depending on your compiler flags you will get different versions of ::operator new being called, and because of the alignment requirements the aligned operator new can't just forward to the default new implementation. So introducing it is the first time failure of a program to implement the full suite of operators results in an actual runtime error vs minor inefficiency.
The reality is that the minimally correct solution is for all programs and libraries that overload the global allocation operators to override all of them. The better solution is for these programs and libraries to stop overloading the global allocators.
Many years ago (talking >a decade at this point) when webkit first adopted a non-default allocator it overloaded the global operators. Perhaps unsurprisingly this caused issues, and now webkit (and presumably blink) do the correct thing: there's a standard base class (FastAllocated or something) that defines operator new, delete, and the [] variants, and using that as a base class results in the non-default allocator.
That's not very nice. The root comment said nothing about making the system handle this automatically, it just described an idea for a potential fix to be applied to this particular case:
> It should be straightforward to make a little LD_PRELOAD shim to implement the new operator new on top of old overloads and thus restore proper functioning.
:(
It was not intended as a dismissive or derisive laugh at the author, but a laugh at the absurdity of the issue itself. Think "haha, you'd think that the reason is X, but technology is involved, and so everythong is terrible" vs "haha you're dumb" which sure as heck was not my intended message.
This is not an easy problem to solve, unfortunately. On MacOS I believe they solve this problem using the two-level namespace: symbol references include the library name, so “operator new(size_t)” from libstdc++ is distinct from “operator new(size_t)” from libtcmalloc.
Symbol versioning also seems like it should solve the problem: have the new interfaces explicitly declared with a newer ABI version (e.g. @@LIBCXX_17) and link only to those new versions from code that expects them. Of course, symbol versioning comes with its own set of nasty drawbacks, but in this case it seems like a solution that might work?
Oh absolutely, there's no guarantee it's ever aligned: the allocator could wrap an aligned allocator but include a pointer sized prefix (a la array allocations) so you would be _guaranteed_ to never be more than pointer size aligned :D
As you say versioning and namespacing is super problematic, but I'm not sure they'd even work here.
At it's core the problem is that some code is compiling with the knowledge it has aligned allocations, so can assume alignment, and the some parts are not. There are a bunch of options that ensure that the allocator is consistent, but they devolve to either ignoring the new+delete overrides, or having the aligned allocators detect the override and forward to unaligned allocators while hoping nothing depended on correct alignment.
Libcxx (the example here) uses posix_memalign for its aligned allocation - which tcmalloc could _also_ have overridden but doesn't. Again the problem is only some of the allocation routines being overridden.
I think the problem here is partly with the implementation of aligned new/delete. Since one is free to override only the old versions, the ones supplied by the standard library should make sure not to fall back to functions that may be partially overriden.
Due to some wonderful C++ features the dynamic linker is forced to unify symbols across shared libraries, even if those symbols have different versions.
This utterly breaks loading multiple libLLVM's except if you build the copy you care about with -no-gnu-unique (or whatever the flag was called)
I have seen wonderful things like the initializers of an already loaded libLLVM being rerun when a new one is loaded.
This isn't something Fedora is doing wrong, unfortunately some games build against older libraries or are built against Debian/Ubuntu and the Flatpak runtimes generally have better compatibility.
Why is this automatically using a new, incompatible solib, instead of a versioned solib?
The author states this is an llvm16 issue, but unless the driver was built against llvm16, it should still be loading llvm15.
If it was built against llvm16 (or loads llvm16), and doesn't work, that's not a failure of anything other than QA testing.
It seems like a big part of the issues is a lack of maintenance. TF2 would actually run better on Linux via Proton but VAC isn't enabled so you can't join the vast majority of servers.
Valve also has existing Source engine tooling that allows Linux ports to drop OpenGL entirely (dxvk-native as used by Portal 2 and L4D2) but they haven't added it to TF2... :(
The fact that these are happening does not necessarily mean the client is getting worse. For example, it could mean that more people are installing Steam for Linux. There is no baseline to say it's getting worse, because nobody opens an issue saying "all working here."
In my experience, the only issue I have on Wayland is this: https://github.com/ValveSoftware/steam-for-linux/issues/7245 (workaround: disable animated avatars) (edit: all AMD machine)
> outside of Steam Deck ofc
There is nothing special about the Steam Deck. It's just another Linux machine.
> TF2
I don't play any Source games, but I could see TF2 having issues because it's in maintenance mode. If it is bjorked that has nothing to do with Steam.
> there is nothing special about the steam deck
How is first party support for the hardware and software stack "nothing special"?
> If it is bjorked that has nothing to do with Steam.
Maybe it's not directly related to the rest of my comment but it's related to the OP. I also think it's indicative of Valve's issues with Linux.
Because the vast majority of that stack (the kernel, GPU driver, window manager, and so forth) has nothing to do with Valve. They might contribute drivers to the kernel (I'm not sure if they actually do - I would expect AMD to be doing that), but otherwise it's an Arch-based distro with the same Steam client and Proton runtime that everyone else is using.
That's not true. It's a read-only linux on a fixed hardware platform, which is a vaaastly different beast than the myriad of hardware/software combinations that exist out there in the wild.
I have heard that argument about macOS a lot, and this is nothing like that. There isn't some "special sauce er... Source" that they apply to their platform. It's just GPL Linux. They may have avoided bad decisions like relying on NVIDIA for Linux gaming, but that's hardly the level of ownership that you see with other vertical integrations. If I use an AMD CPU (or Intel, which would be arguably better) and AMD GPU, there is no reason why my PC couldn't be just as "first-party" as the Steam Deck.
Wine/Proton ultimately access the GPU through DRM, that remains the same for Valve hardware or custom-built hardware. Both Steam and Wine/Proton currently render via X11 (via XWayland if necessary), on both my PC and the Steam Deck.
I feel like there is a gap of understanding how a HAL works here.
When you have a platform like Steam Deck, it's the platform that gets tested by QA, and the platform that most of your devs are building for every day.
In the gamedev world I heard a lot of people not wanting to support linux because they never know which glibc version to support, which mesa version to support, which hardware GPUs to support, which graphical API to support, etc.
Cutting down that matrix (and I just mentioned the most egregious examples) to only one element is invaluable in ensuring your users have a bug free experience.
"Just" GPL Linux encompasses myriad library versions, kernel versions, driver versions and varied hardware.
> I feel like there is a gap of understanding how a HAL works here.
Just because you have a HAL doesn't mean that you don't get different behavior and crashes with different numbers of CPUs/concurrency or other hardware beneath. Modern GPUs are also pretty complicated beasts, and assuming that's fully abstracted is a mistake.
And this all leaves aside the myriad of other problems you can have with the ensemble of software running on the machine that interacts with the game (directly or indirectly).
Being able to test and make one restricted platform work well is a far different beast than covering the huge mass of variation users create on their own machines.
I’m worried that Valve puts too much resources into Proton (derivate of WINE) instead of tooling for native ports. Yes, Proton is needed to provide initial compatibility. But Proton is another layer of complexity (more bugs, integration, system resources) which requires more programming. I started playing CS again after it was ported natively in 2014, it runs well and all issues with WINE were gone.
If Proton becomes to “good” we end up in a situation with a high maintenance burden for Valve. Game developers will rely on it and Valve has all the constant work. Instead game developers should treat Linux as first-class platform for AAA-Titles, for which the need appropriate APIs, compatibility and tooling. As Valve does itself support Linux as first-class platform from HL2 to CSGO. The target shall be official support from the very first day.
Anyway. Looks like Valve has chosen a special implementation for TF2? What I miss here is a link to a bug report. Ideally opened months ago :)
Besides that, PlayStation OS is based on FreeBSD. Even if the 3D API is different, it is just yet another backend.
They don't port them, because the QA and support aren't worth the sales, that is about it.
Edit: More details in https://news.ycombinator.com/item?id=34634854