The ABI status of ELF hash tables
lwn.net
lwn.net
The entire premise of client side anti-cheat tools is, "trust the client." I think pretty much everyone at Hacker News knows that's the exact opposite of how you make a secure system.
Never trust the client.
So at this point, the maintenance burden on glibc is however much effort they want to spend justifying breaking backwards compatibility.
Fundamentally the only way to provide the kind of compatibility you're demanding, where any level of reverse engineering or parallel implementation must be supported in perpetuity, is to just Never Change Anything At All.
[1] With the sole known exception of the "runtime linker" embedded in this EAC product under discussion.
Never Change Anything is far from the ask here. The ask is that when real actual breakage is found, the benefits of keeping said breakage justify not spending the engineering costs to fix backwards compatibility. Near as I can tell, the benefit is 16kB disk space savings, and glibc's engineering cost to maintain compatibility is nothing.
Like, another backwards compatibility breakage I saw recently was Graphene switching from 39-bit to 48-bit virtual address space, which also broke games. That has a real measurable security benefit, and actual engineering costs to develop a backwards compatibility story. But they still spent effort to fix breakages.
No, that's simply wrong. No "glibc binary" that could have been even theoretically compiled within the past few decades will fail to run because of this issue.
What happened is that this EAC product had reverse-engineered the internal formats glibc uses to provide the APIs it implements. And that format changed.
I repeat what I said elsewhere: it is simply not possible for any software product to anticipate all possible external users of its internal implementation details. What you want, you cannot have. (And I repeat what I said elsewhere too: you don't have this, even on windows. Microsoft breaks things like anti-cheat hackery with every release too, it's just that you don't notice because the proprietary vendors care enough to fix it, where they don't on Linux.)
That's obviously not true. The entire story is how a new glibc binary broke existing binaries.
> What happened is that this EAC product had reverse-engineered the internal formats glibc
Okay I missed this; I thought EAC was parsing the libc binary as per ELF. What was reverse engineered?
> it is simply not possible for any software product to anticipate all possible external users of its internal implementation details
Correct. Which is why I never asked for stuff that never breaks.
What is important is that when stuff is found to break, the breakage and refusal to provide backwards compatibility has a compelling reason. glibc's justifications here are laughable; they're spending more effort to not revert the breakage than it would take to fix.
Which is kinda why the story blew up.
No, you got it the wrong way around. EAC has used the well-documented and SysV-standard DT_HASH mechanism.
The DT_GNU_HASH is the (almost) undocumented one that they would have to reverse-engineer from GNU ld's source code
The only things that can be done in those cases are loosely "detecting" cheaters through heuristics and human review of suspicious behaviour, which won't catch more careful cheaters and may lead to false positives, and client side solutions like EAC which frustrate less skilled cheat developers and sometimes catch the more skilled ones off guard. Alternatively, give up on PC and focus on consoles which are sufficiently locked down that cheating is currently impossible regardless of how much you trust the client.
Great talk on it here: https://www.youtube.com/watch?v=kTiP0zKF9bc
I can say that Valorant's anti cheat is the singular reason I've never tried it, even though I love counter-strike and similar games. Giving game devs (especially those owned by Tencent) ring 0 access is, uh, a step too far I think.
The ship has sailed on ring0 really, all of the major anticheat players plug into ring0 on Windows. If you play online games on Windows there's almost no avoiding it.
Interestingly Proton acts as a loophole, EAC and BattlEye don't touch ring0 on Linux even though they do on Windows. Gamedevs have to opt-in to allowing Linux clients though.
If steam deck is successful enough I wonder how long until kernel modules become a requirement...
Yeah, there's already Linux-specific cheats in the wild which abuse the fact that the Linux anticheats are oblivious to anything happening outside of the current user, so they can manipulate the games memory with complete impunity simply by doing it from root or a kernel module. Developers may become more hesitant to allow Linux clients as abuse becomes more common, unless the anticheats start extending into ring0 on Linux the same way they do on Windows.
I seriously hope these corporations don't start pushing this on Linux gamers. They need to accept that they can't control it and move on.
Offload too many tasks to the server and you end up with a game where you encounter latency where it's critical, like hit detection (and likely many others). Those extra milliseconds does make or break the experience.
Not saying it's an impossible problem to solve, just that multiplayer games are operating in a time-sensitive environment.
> Blaming the EAC developers for not keeping up with Linux ELF hash-table formats might not be entirely fair. The DT_HASH format is mandated by the System V ABI specification, the DT_GNU_HASH format is undocumented, and there has been no deprecation campaign to get users to move on.
Most problems like this are communication and co-ordination problems.
Which only exist in this case because the software isn't open source. This would have been spotted years ago in open source.
Of course it goes almost without saying that EAC would make about as much sense as an open source project as it does as a closed source project, which is to say none whatsoever because it's conceptually nonsense :-)
Which is obvious once you notice there is an active "anti-anti-cheat" marketplace.
Personally, I feel that by the time you're messing with the _DYNAMIC entries of an ELF file, you need to be prepared to actually follow and keep up with ELF handling rules. Deleting DT_HASH feels no different than, say, changing the format of the structure pointed to by DT_DEBUG, which happened just last year (https://sourceware.org/bugzilla/show_bug.cgi?id=15971, https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=a93d9e03...). Which, you know, breaks debuggers--but no one's running around saying "oh no glibc's ABI story is complete and utter trash!" because of that change (I daresay most of the people commenting on this issue aren't even aware that it happened).
This is especially salient given that the DT_GNU_HASH format is not even documented. Leaving it as the only-option-by-default going forward is unconscionable.
It seems to me that EAC includes its own copy of a dynamic linker for the express purpose of validating that glibc functions are not obviously detoured in the current process.
This would explain why people are so annoyed: there's unplanned extra low-level work to enable glibc upgrades two years in a row.
Closed source software that wants to hold back open source software can go pound sand. Gamers can run their games with older glibc in a chroot / container. I say this as someone who does exactly this for games.
BTW my comment lists two ways for people to continue "still actually USING their computers".
I dont think the change was inherently bad but just ham fisting it into the release without widely announcing that the ABI will change is a massive failure on the glibc teams end.
It's unfortunate, sure, but it's just not possible to prevent. And all the people screaming about "This Doesn't Happen On Windows" are just fooling themselves (or, to be blunt, being deliberately disingenous). This happens on windows every single release. Windows N+1 comes out and suddently a bunch of hacky drivers or tools or gadgets stop working until their authors manage to figure out what part of their trickery Microsoft broke.
Yes, I know Linus Torvalds says "don't break userspace," but that's for officially exposed kernel interfaces.
https://news.ycombinator.com/item?id=32471624 ("Win32 Is the Only Stable ABI on Linux?", over 500 comments)
What works compatibly[1] is the win32 C API implemented along with the specific set of DLLs that you built your application against[2]. MS curates the kernel/driver interface below that fairly well[3]. But the kind of trickery EAC is doing here is right in the middle of things that absolutely do not work on windows either.
[1] In the sense of "do this and your app won't break".
[2] Which is why windows binaries ship their own C libraries. FWIW this model exists in Linux too. It's basically the way we package Docker and Snap containers. Had EAC been written against a particular Snap environment, we wouldn't be having this problem either because it would be frozen on a particular glibc version.
[3] And above that level, there are often other runtimes like .NET to which you can portably target.
step 1: provide complete and accurate documentation for the new format and communicate it widely and prominently to the developer community
step 2: set a target date & version for removal, and get feedback from the community to ensure it gives sufficient time and information to complete a migration. Provide a way for developers to ask for and get support to make the necessary changes. Release sample code, if possible.
step 3: one release before removal, loudly note the deprecation and the upcoming change.
step 4: as the release that removes backward compatibility is being prepared, again, communicate the upcoming change and provide guidance for those who can't update on any ways to mitigate problems.
step 5: finally, release the version with the breaking changes, and put the break boldly at the top of the release notes/changelog. Reiterate the previous guidance on mitigating problems and help.
step 6: Go have a tasty beverage.
I strongly suspect that the reason for the "abrupt" removal was that it wasn't expected that anything would break: that table is used for dynamic linking, and the dynamic linker comes with glibc itself, so it can be assumed to understand the new table (and the dynamic linker which comes with other C libraries would be used to load these C libraries, and not glibc since you cannot load more than one C library in a single process). AFAIK, the code in the dynamic linker which understands files with only the old table wasn't removed, so from that point of view, compatibility was being kept.
That is, they didn't expect anything to be using that table without going through dlopen() and dlsym().
You actually can, and there are even some benighted corner cases where you need to (eg, providing a fake glibc to a DRM/anticheat system or other such malware while your plugins use a different one). It's not properly supported and making it work is !!fun!! in the Dwarf Fortress sense.
It was replaced by DT_GNU_HASH in the gnu implementation of libc
....I think these people have forgotten what GNU stands for.
Maintainers of a package used by literally almost every executable really ought to use the latter thought process, IMHO. Hiding behind the "well, technically..." is kind of irresponsible. Hiding behind "It's been 16 years since we introduced $NEW_THING" is doubly so. Who the hell reads glibc's changelists for fun?
You know the quote: "on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.`"