To save C, we must save ABI
thephd.dev
thephd.dev
The actual most painful part of ABI breakage is if you have, say, library A that's compiled against old-ABI and library B that's compiled against new-ABI and library A passes ABI-varying types to library B. Like, say, std::string (which changed ABI as a result of C++11). gcc managed to make it work--but the process was so painful (and required adding new features to C++ to do so!), that they have said that they do not want to have to ever do that again.
It's relatively easy to completely change the functionality of a library function and use this sort of magic to always select the right function. But when you broach changing something like intmax_t and time_t, you have to be cognizant that the issue is different libraries (neither of whom are the standard library) who aren't prepared for the type to be different communicating with the same type.
https://thephd.dev/to-save-c-we-must-save-abi-fixing-c-funct...
The author then argues that it is at least a step forward:
> For those of us in large ecosystems who have to write plugins or play nice with other applications and system libraries, we’re generally the last to get the benefits. But, at least we’ll finally have the chance to have that discussion with our communities, rather than just being outright denied the opportunity before Day 0.
This is the core of the issue. I continue to maintain that these issues are not insurmountable, bit we have a "people problem" that makes them effectively insurmountable. There are too many people with deep expertise in one area of C++ (compiler implementation, language design, etc.), who don't have a big-picture view, yet will let their opinions known as though they know everything. And it's not just C++, all open-source languages have this problem. Design-by-committee never did anyone any favors. It's precisely WHY we keep getting half-assed implementations like regex and nested functions approved in the standard library pr GCC extension, and it's WHY no one is ever permitted to fix them.
The only solution is for people to abandon their egos, admit that someone may know more than them (choc et horreur!), and try to do what's right for the users.
So I don't think your criticism is valid in this case. It's more a general problem with all language extensions, and how they interplay with language evolution.
Out of interest (I'm not tracking the C++ standard much these days) such as? For example I know about the order of struct members and bit fields, but that has always been the case in C, and so in C++.
Bitfields are profoundly non-standard.
edit: It's under "Bit-Fields" in [1]. I'm not entirely happy with the wording, but it's there.
[1] https://raw.githubusercontent.com/wiki/hjl-tools/x86-psABI/x...
Two hours later, I gave up. The mysterious side-by-side manifest error wasn’t solvable. I installed all the redists, and even installed a neat redist installer I randomly found on GitHub which brute-force-installs every possible redist known to humanity.
Didn’t matter. The side by side error was just insurmountable.
Now, as a former gamedev who grew up on windows, I have a soft spot for Microsoft. I love me some visual studio. Yum yum, that IDE was cool a decade before Jetbrains showed everyone why IDEs are cool.
But it’s been… eight years now since I switched to macOS, and more generally exited from the MS development ecosystem.
I don’t know, man (or woman or they). Microsoft’s current policies are hopelessly complicated compared to unix. I’ve been programming for almost two decades. If I can’t analyze and fix an error that clearly shows exactly which manifest entry is missing, well… I think we’ll just agree to disagree that Microsoft is making nice decisions about ABIs and related ideas.
Here’s a wild idea: just let a program run, and trap errors into an error log. Jetbrains does it. Their early access IDE happily lurches along even with dozens of Frankensteinesque errors coming up.
Refusing to start because of an ABI mismatch (or indeed, dying at the first error) was one of the most unfortunate decisions in OS design.
Probably not so great for security. I’d expect most cases where Windows switched from being lenient to being strict are exploit mitigations.
And many Windows developers choose to rather install the specific minor version they used globally, instead of just vendoring it in the program folder (even though the license allows it, hence "redistributable").
Right now my machine has no less than 4 versions of the 2008 redist installed globally, for each x86 and x64.
Nothing about this or the way redists are managed seems hopelessly complicated. Also there is nothing it can do but die at the first error in this case, it's not side modules or components failing to load it's the stuff like memcpy and iostream for the main process of the app.
Why mess with it at all? It's portable assembly. Why are we trying to use it as more than that? Even the kernel doesn't need it or only needs tiny bits of it. See Redox OS and all the other similar projects.
Why do we still talk about ABIs? We can statically link a whole binary, and still use far less disk and ram than we do now with containers.
We have high level languages that "link" at the level of names and modules, not bits and bytes.
C works well enough for what it's used for. Rust and the others are up and coming.
Instead of saving C, why not improve Rust and JavaScript and the others?
At your level and mine, it looks like portable assembly. At the level of the OP and the compiler implementations, it is absolutely not.
> Instead of saving C, why not improve Rust and JavaScript and the others?
"Platform effects" is a pretty complete answer. C isn't going anywhere soon. You and everyone here depends upon a mountain of C code every day.
There's a few possible transition paths though. Like a backwards compatible strict superset or conversion tool.
The sooner other languages mature, the sooner we can stop writing new projects that use large amounts of C.
If we incrementally evolve like that, it might take 50 years to get to the kind of environment that Rust seems to be aiming for, where everything is safe and high level abstraction are cheap and available anywhere, and we have official package management.
ABI is a necessity for FFI.
[1] https://www.akkadia.org/drepper/symbol-versioning
ETA: Hmm, maybe semantic interposition / the global namespace of ELF dynamic linking is at fault here? As in, if a shared object declares an import of printf and a glibc version, it should get printf@@GLIBC_whatever if the import actually ends up being from glibc but plain printf otherwise? Not sure that’s a good interpretation of what versioning should mean in this context but it’s a possible one that would force this unpleasantness.
It's really awful to hold back a language because a few companies cannot deal with it.
C++ should really start to deprecate thing, and reduce backward compatibility.
Not to mention there are probably a few ways to use old code and to wrap it.
It sort of boggles my mind how new C++ versions were able to maintain backward compatibility. Compilers must have a hard time doing it.
That's why I think C++ should be forked... Not to mention microsoft is probably the main culprit here.
In fact GCC (and indirectly Red Hat and the other Linux distros), have been the most vocal for not breaking the ABI as many still have the scars of past ABI breaks .
The big difference to UNIX shared objects (with exception of Aix), is that Windows dynamic libraries are namespaced and symbols are private by default.
That makes huge difference.
Yes, the other big difference is that every DLL has its own namespace, so you can actually load two CRTs into the same process just fine - so long as you don't pass pointers/handles from one runtime to the other. But this kind of multi-runtime arrangement doesn't happen often in practice, except in plugin/extension scenarios (where different extensions might bring in different runtimes into the same process).
Even if they aren't installed on System32 (thankfully), they get their own slot under Programs.
Most applications that do xcopy installs tend to be .NET based.
Some platforms have a documented ABI across the board which covers all their APIs together with the principal compiler family.
Others, like Windows, have an ABI for some core system interfaces. Development tools can have different ABIs in their ecosystem.
The rest of the article seems to be the complaint that there are problems linking code from different compilers on Microsoft Windows*, which is completely unsurprising in the light of my above remark.
The fix is: don't mix code from different compilers on Windows without adding the calling convention declaration specifiers (an language extension) and ensuring these are in the header files that are mutually used.
It’s more than that: individual compilers can’t evolve (for example change the size of intmax_t), and therefore certain desired ISO C features couldn’t be implemented, because that would break ABI compatibility with code compiled by earlier versions, which would prevent users from upgrading their compiler until all their dependencies have been upgraded, which (a) may never happen and/or (b) may be a newer version with other compatibility breaks. It may be okay in a everything-is-open-source-recompile-the-world setting, but in general it’s absolutely not practical.
That is mostly a myth, except maybe in some embedded situations.
> individual compilers can’t evolve (for example change the size of intmax_t)
Compilers can provide __my_intmax_t which is decoupled from the standard intmax_t.
If there are system libraries and third party libraries using intmax_t, you don't want to change it; that isn't really a valid form of evolution.
Long before 64 bit systems were common, compilers had local types like __int64_t. Programs could detect (or assume) their presence, and typedef them to something nicer.
GCC has __int128_t on platforms where intmax_t is 64 bits.
That either means breaking compatibility with existing sources (which have to be changed to use __my_intmax_t instead of intmax_t), or not conforming to the standard (if __my_intmax_t is now the one with the semantics of ISO C202x intmax_t). Compiler vendors want neither.
Btw the sample break is between two long long and one int128, why they have to the same I wonder.
I want to fix something in it, so I need to fix C++, C, and the gABI. Three separate processes, each needing about 3 years, though C and C++ are aligned now.
That said, I think this idea is great, especially the fact that literally the libraries already craft their own labels, this just makes it a part of the language.
Also, as the video states, the issue can happen even if the type of one of the arguments is the same but the fields in the type are rearranged, leading to a different offset during derreferencing.
For example, IIRC, ELF dynamic linking occasionally forces the compiler and/or linker through spectacular contortions in order to ensure two pointers to a single function compare equal (as required by the C standard and as happens with static libraries) even if the values being compared originate from two different shared objects, the function itself resides in a third one, and the compiler has no idea which functions are external to the shared object its output will be linked into.
(Windows tells the standard to go take a hike, or more charitably doesn’t attempt to badly emulate the semantics of static linking in dynamic linking. I’m actually with Windows here, even if their executable format is a horrible underdocumented mess.)
Those contortions are not required when the function is only ever called and never has its address taken, except compilers can be pretty naive about that part. Not that they have to be, but at least it’s not an obvious sure thing.
Also makes life difficult for the optimiser; I think solvably as long as the new pointer is indeed static.
#include <stdio.h>
typedef int int_func(int x);
void print_result(int_func *func, int x) { printf("%d\n", func(x)); }
int f(int x) { return x; }
_Alias f_alias = f;
static int_func * const f_ptr = f;
void doesnt_work(int x) {
print_result(&f, x);
print_result(&f_alias, x);
//print_result(&f_ptr, x);
//error: incompatible pointer types
}
This is what _Alias is meant to solve: it's perfectly transparent, in that the end-user has no way to distinguish whether the name comes from the original declaration or an _Alias. This allows libraries to modify the external name of a function, without breaking code that depends on the syntactic name of the function.
#define f_deref_ptr (*f_ptr)
/* ... */
print_result(&f_deref_ptr, x);
Also, there's already the possibility that someone has defined a struct containing two ints to act as a 128bit integer, intmax_t is already sometimes smaller than the largest integer type.
Is intmax_t supposed to be the largest integer in the standard or supported natively by the platform? If it's the second, leaving it unchanged when introducing larger ints wouldn't be a problem.
A struct of two integers couldn't be used in regular & bitwise arithmetic, added to pointers, index arrays, cast up and down to other integer types, etc.
As-is, you can pass in & out intmax_t as a default case and, worst-case, you just waste space. But "uint128_t a = ~UINTMAX_C(0)" not making a variable of all 1s bits would be just straight up broken.
On Windows, every DLL with a different name also has its own distinct symbol namespace. Thus, the conflict you describe can only arise if your code explicitly propagates some time_t* value from one library to the other.
Could someone explain what they're actually driving at?
> Just a type change! Shouldn’t change the assembly too much, right?
What reasonable person would think that? You're changing something from 64 bits wide to 128 bits wide. Of course the compiled code from the 64-bit version is looking at a single 64-bit register, with the 128-bit version looking at multiple registers. Why is this unexpected?
Who would ever expect that you could just change the width of the inputs and return type of a function and expect it not to break the ABI?
> Okay, so in C we can break ABI just by having the wrong types on a function and not matching it up with a declaration
Of course. Why wouldn't you think that?
> What if I told you this exact problem could happen, even if the header’s code read extern long long do_stuff(long long value);, and the implementation file had the right declaration and looked fine too?
Alright, I'm almost intrigued enough to keep reading. I hope they get to the point soon though.
> [pages and pages about linux package maintainers and red herrings about python and C++]
> [going on and on about "the problem" without having ever made it explicit]
At this point I give up trying to find the author's point. What needs to be saved about the ABI specifically? I haven't found the problem they're talking about and I've given up trying to skim for it. I certainly am not going to read this whole thing verbatim because they're just too much time-wasting cruft here.
If the answer is “the ability to change the types of library functions without changing their name” (which is what his first few examples were showing to be the “problem”), then of course you can’t do that, and I’m not convinced we should waste any time or effort trying to get that to work. If you want to change the types accepted/returned by a function, make a new function with a new name.
Of course, if there’s something I’m missing here I’d love to know… the article has done a terrible job outlining it if that’s the case.
std::move_fast_and_break_things::vector
Or maybe the other way around so you can keep using std::vector: move_fast_and_break_things::std::vector
Then you can add a 'using namespace move_fast_and_break_things;' at the top of your files if you know you don't care about the ABI in your program.
Abseil doesn't even have releases any more. They expect you to live at head like GOOG does internally.
For how transparent aliases help solve this, suppose that there is some ancient library function called get_year:
typedef int32_t time_t;
int get_year(time_t time);
// document time_t and get_year
typedef int64_t time_t;
int get_year_v2(time_t time);
_Alias get_year = get_year_v2;
// document time_t and get_year
I think it’s a useful tool but I don’t think it’s an ABI versioning panacea (even c++ adoption of inline namespaces within libraries is limited with the standard library being the only place I’ve seen it used in a meaningful way).
You don't need LTO for dead code elimination.
And the new code adopts get_year_64 directly and you don’t blow up system complexity with features that you don’t really need.
Old functions should not have first mover advantages on names, nobody wants to riddle their function calls with *_v2 all over the place, and neither do library maintainers want people to keep using *_v2 when *_v3 fixes issues present in *_v2.
I agree there's a lot of fluff in the article but complaining even more when someone goes out of their way to appease you is just too much.
He is also the author of sol2, arguably the best C++ Lua bindings library out there :-)
His writing style might be a bit idiosyncratic, but he certainly is a brilliant person.
C is already everywhere. And so unsafe, the sooner it dies, the better
What we should be saving is Pascal.