ABI vs. API (2004)
lists.debian.org
lists.debian.org
It's a pity that Colin Watson left Debian's Technical Committee (https://lists.debian.org/debian-ctte/2014/11/msg00052.html), but his blog is still worth reading: http://www.chiark.greenend.org.uk/~cjwatson/blog/
https://community.kde.org/Policies/Binary_Compatibility_Issu...
Check out the "fragile base class problem" some time for more things than can go wrong.
Something that want made explicit: If you support closed source, shipping binaries, then the ABI can't change, or you break all compiled binaries users have installed. If rebuilding the world from source is an option, changing the ABI is an option.
(Introducing a new ABI for new subsystems is still OK - hence renaming libc when it changes)
(Say what you want about Microsoft, but COM solved that problem pretty well in the '90s...)
It's an option ... but not a very good one.
It tends to create an unstable environment that requires a centralized organization just to keep it working, precluding a healthy diverse software ecosystem.
This is a major reason why it's very difficult to distribute binary packages when operating externally to the distribution packagers, and why commercial software distribution on Linux is a massive pain in the ass.
Although desribed in terms of networks, it is flexible/abstract/content-free (1) enough to describe communication between software components on a single machine.
An API describes what programs/libraries say to each other, an ABI is a lower layer that describes how they should do that. I would place them on levels 4 and 3. Level 2 is the hardware in the CPU, level 1 are electrons moving through tiny wires, but one can change that to something different (light flowing through fibers, a human keeping track of state on paper, etc) without affecting ABI nor API.
Looking at this this way, it is clear that there are levels above an API. For example, one could place the semantics of the API (you can only close a file that is open, free memory you allocated, etc) at level 5.
(1) I think it is a mix of these, but feel free to pick an adjective you like.
The third way of carefully doing only additive modifications to the library to preserve ABI even over upgrades is hard - that's true. But I don't think it's needed that often once most of the features are in. And again - it applies only if you've got some really popular software that gains anything by doing it this way, rather than just bumping up the version.
So with shared library dependencies, you can be very broad (e.g. "any libm.so.5 please") and get security updates etc. automatically without needing to recompile, while in "modern" programming languages you will be locked onto old and rancid libraries because the app developer doesn't bother updating the locks.
Related: I'd like it very much if compatibility was enforced automatically. For example, if I publish version 1.2 of my library to $repository, then $repository checks if 1.2 passes the 1.2 tests (of course), and also if it passes the 1.1 and 1.0 tests. If any of these tests fail, then release should be downright refused unless you upload it as 2.0.
And of course, once you have that, you can think of any check you like. In C++, you can pretty reliably detect code changes that change your ABI in backwards-compatible or -incompatible ways, so you could allow backwards-incompatible changes only in major and backwards-compatible changes only in minor releases.
And the recently introduced container based replacements are and exercise in shooting twee twee birds with AAA...
Eventually you drop the old one, because you can no longer provide security support or for some other reason. But if there are compelling reasons otherwise, distributions can and do provide multiple ABIs for the same library.
Thing is that with sonames the actual files are distinguished anyways, so mangling the package names to get around collisions are just an artifact of an overly rigid package manager.
How do I ensure that I conform to these rules apart from being disciplined about them? If everything compiles, am I good? No! If everything links, am I good? No! Depending on linker options and the nature of incompatibility the program using incompatible ABIs can blow up at runtime or just silently corrupt data.
And don't get me started on the venerable "pimpl idiom". A page of boilerplate just to ensure the most basic thing.
Hope that clarifies my short sentiment a bit :) I agree that once you grasp the rules following them is not that hard but it is just another bit of incidental complexity that we agreed to maintain.
Of course it won't save you from business logic bugs (changing the meaning, but not size/location of a value), but "how do I ensure that I conform" is 90+% possible to verify automatically.
This problem is solvable at the language/runtime level by resolving vtable offsets at runtime, but that doesn't help the existing C++ userbase.
Most people just have to incorporate their C++ dependencies directly instead, as there is no supported ABI.
It's sort of like democracy (or capitalism?): the worst system in the world, except for every other alternative. :)
Having said that, many of your pain points are specific to C++ rather than system-level ABIs. C++ compilers have to abuse most OS-level system ABIs (example: symbol name mangling) because of linkage-related concepts in C++ that don't exist at the lower-level (and simpler, yet very different) system ABI provided by the OS.
In a way, you're really complaining (rightfully so, IMO) about one of C++'s core design principles: that the programmer should never pay a performance cost (compared to C) for language features that aren't used. For example, methods are non-virtual by default. Support for virtual methods are required for polymorphism, one of the defining features of OOP. However, calling a virtual method is always going to have at least a teensy bit of overhead vs. calling a non-virtual method. As a result, in C++ all methods are non-virtual unless defined otherwise.
Higher-level (and admittedly <= C++ in performance) OO languages tend to make everything virtual by default, and may not even have a mechanism for making a non-virtual method. These languages give away a small amount of performance in exchange for a reduction in cognitive load on the programmer. Reducing cognitive load is also a reduction in potential bugs, so there may be pragmatic reasons for using a higher-level language other than making the developer's life easier.
Most (but not all) other OO languages take away the option of direct memory management, and instead have some variant of garbage collection to keep memory usage somewhat constrained. C++ can't do this by default because then it would take a performance hit compared to C. So again, you get stuck with one (or two or many) options for higher-level memory management, but by default you're doing manual memory management a la C.
I can see (heh) C++ being very appropriate for certain types of projects and certain types of developers, but IMO its popularity is mysteriously much greater then its usual level of appropriateness. I don't dislike the language (OK, maybe a little), but it's unfortunate that it is considered the default alternative to C as often as it is.
But it is impossible in general.
ABI describes things like how the stack works, but at the assembly level the only stack is the one you implement, which is generally compatible with an ABI version.
And there's still more than one stack implementation choice at the assembly level. For example, what is the correct order for passing parameters on the stack? What is the memory alignment for off-size stack parameters? Is the caller or the callee responsible for reclaiming the stack space at function completion? How are function return values passed back?
At runtime if you're reading/writing to memory you had to ensure that you read and write using the same "stride" ie reading and writing back in 32bits is fine, but if you read in 32bits do some transforms in a register and write it out in 16bits you'd have to make sure the internal register BE representation and the written out to memory LE representation made sense.
The GCC and other compiler guys can give you lots of war stories about SIMD (VMX/VSX on POWER)
Anyways, just something I thought off the top of my head :)
btw: so are you advocating a full GPL-like distribution of the sources alongsides the binaries for all programs?
Not the parent, but I definitely advocate for that, so long as it maintains user freedom.
I only know that windows does something different more akin to debug symbols in each binary by default with chicanery to not have it impact things in the general case. But I'm not a windows programmer this is about as much as I know about windows in that regard.
And that leaves no current viable alternatives to the Unix-like systems. Sad, really.
What made Unix a success is a great mystery to me. Presumably it was the ability to run a time-sharing system on low-cost hardware, but then people started taking the tradeoffs it had to make as gospel, which is why we have this ridiculous situation of still wrangling processes on computers with 64 bit bloody address spaces.)
> Changing a library's API in a backwards-incompatible way is in general bad form because it means that developers of programs using the library have to change their source code to port the programs to the new library. However, it does happen sometimes in the case of major libraries, such as some of those in GNOME and KDE.
So only the big guys are allowed to break the APIs? Really?
Plenty of libraries break API/ABI all of the time. Notoriously FFMPEG was/is terrible about it. But depending on libraries that are hugely unstable in this way is terrible, so in most cases, people go out of their ways to avoid doing so (most frequently using another library, or in harder-to-avoid cases creating an interposer library to smooth out the instabilities of the underlying library).
OSS developers probably know more than anyone in the world that the most expensive thing to do in software engineering is maintaining a piece of software as time moves forward. Anything that makes that job more difficult needs to have an enormous upside to make it worth while, and in almost every case, using an unstable library is just not worth the extra maintenance costs attached.
I think what he meant to say is "Even major libraries do it sometimes." It is already understood that minor libraries do it.
A colorary to the above is: libraries that break their API/AIB too early or too ofter never get a chance to earn the 'major' label in the first place.
You could also argue that a major library that abuses their privilege and breaks backwards compatibility beyond their "karma ballance" will be eventually forked or otherwise replaced by a more stable competitor. I am not sure if this idea holds water in the real world, but I really hope it does.
Little inconsequential libraries can move fast and break things, though there's a peculiar paradox here. As the number of dependencies grows, the tolerance for change decreases, yet in terms of perception, little libraries that aren't major dependencies that change frequently look untrustworthy and are liable to never be trusted because of their flux.
That is, if you want to be successful, be predictable and consistent even if that means being leaving some potential unrealized.