ABI compatibility in Python: How hard could it be?
blog.trailofbits.com
blog.trailofbits.com
One day the release pipeline broke because a dependency's authors published a new version containing a wheel for a newer ABI, though the older ABI was still supported. My build pipeline pulled the one for the later ABI because it was compatible, but the release pipeline environment, which could not be upgraded, did not have the newer version of gcc required. It was a nasty introduction to the fact that one Python package version can have multiple published wheels targeting different C++ compilers!
But the worst part was not that I learned something new! It was that I wrangled with both pex and pip for a few hours trying to figure out how to download a wheel for an older ABI as both tools resolve the latest compatible by default. The options to do so are ostensibly there but the effect was not. I definitely could have made some mistakes but in my mind it shouldn't be _hard_ to get it right.
I ended up downloading the wheel for the internal package in the build and copying it directly into the artifact. The release pipeline resolved public dependencies and installed the internal package from the local file.
I admit to being unaware of "versions inside versions" where a version may have multiple published ABIs that are not compatible across systems, but a nice packaging system would still make it easy for me to use platform X to build for platform Y.
Erm, is this not the failure of C++ standards instead of Python?
This seems like Python is the failure point being caused by C++ not having a stable ABI. Python is simply trying to paper over the brokenness of the C++ ecosystem.
Am I missing something?
The package I was using had multiple artifacts per version, including the raw zipped source code and wheels for each ABI they built for. It’s certainly convenient that the most applicable one for your current system is pulled automatically by pip. But in this case that’s not the behavior I wanted, and I could not successfully get pex or pip to download a “less optimal” artifact.
You can blame C++, but if Python is already papering over deficiencies it’s not unreasonable to expect improvements that aren’t fundamentally impossible. It should definitely be possible for there to be one easily understood argument to specify which artifact pip should download. In my (potentially flawed) experience, figuring out which ABI value to pass was already not straightforward. Even then, the arguments for selecting an ABI or even asking for the raw source instead of a wheel weren’t seeming to have an effect.
Python is simply trying to paper over the brokenness of the C++ ecosystem.
Pretty much every other ecosystem (except perhaps Javascript?) does this. Look at FreeBSD packages, the ABI is used as part of the identifier. GNU Autoconf, rvm, homebrew, rust, etc., etc. all use and expose an ABI identifier so that you aren't accidentally going to mix and match things.
Trying to use Python in Nix is frequently a bother, but at least once it works, it works.
1. Build the artifact that you depend on, publish it on "pypi or something" referenced by both the version of the library and a hash that uniquely identifies the build. This is the approach taken by conda, for better or worse.
2. Only allow one canonical build of a library you depend on so that the "version" becomes a unique identifier for the build. This is the approach taken by distribution package managers.
3. Create metapackages that describe ABI constraints that are required by packages and must be satisfied by the underlying system. For example, a "cxxabi" package could be provided by the underlying system, and the packaging tools could automatically add dependencies on "cxxabi" at build time, based on either an exact pin to the library built against, or in some cases, a relaxed dependency by inspection of versioned symbols used by the binary.
4. Statically link all your dependencies and/or vendor all your dependencies. These are used by quite a few pypi packages that depend on standalone C libraries to avoid most of the issues altogether.
[1] For instance, you cannot version a C or C++ library build independent from the versions of all of the transitive dependencies of that library (more or less). Of the options listed here, only distribution package managers can really account for this problem, and not every distribution package manager cares to.
Wait until you have Rust packages which include and wrap hundred of thousands of lines of very old C and Fortran code like numpy has.
So far Conda is pretty much the only problem I've had with native code mingling with Python (although pip was able to get numpy installed on my BSD box). Otherwise it's an issue of the tooling just being really archaic. Which is amusing to me given how much hand wringing there is over providing one right, idiomatic way to do things (e.g. ternary operator). For me digging into Stable Diffusion (and related bits and bobs) was a bit of an eye opener with just how much cargo culting is going on. Installation instructions are often copy and pasted from other related projects. Or maybe just check out our repo and follow the instructions from some other project.
Honestly, my hope is that some of the effort spent obsessing about language details gets spent on coming up with a better packaging and dependency management story.
Dozens of other languages that are tiny and new do not have good packaging story. Rust does, because it cares about developer experience.
> Wait until you have Rust packages which include and wrap hundred of thousands of lines of very old C and Fortran code like numpy has.
I don't know about Fortran, but C is basically legacy at this point, it doesn't change much and even if it does, there is one some-c-lib wrapper package that takes care of all those issues, Rust apps rarely depend on some C dependency directly.
Rust apps rarely depend on some C dependency directly.
Thing is Rust makes this pretty well invisible to most users because it all works with so little teeth gnashing.
Since it's the same algorithm, most of the tricks appear to make it through the language conversion, so it's reasonably good even on a completely different architecture.
Perhaps, but not directly. It will depend on some libssl-rs which depend on libssl-sys which handles C bindings, so that application authors don't think about it.
"The PyPA publishes the Python Packaging User Guide [1], which is the authoritative resource on how to package, publish, and install Python projects using current tools."
[1]: https://python-poetry.org/
Edit: Some quick cursory research seems to indicate I'd have to do some custom scripting to integrate SWIG into the process.
IMHO Python should improve what it already has instead of pushing the usage of third party tools for a core component such as packaging.
For all my Python projects I just use setuptools and the built-in venv module for isolation.
What I feel it's really missing is an official way to have multiple Python version installed.
I also suppose that pyenv and poetry exist and solve that to a reasonable extent.
Debian IMHO has the worst take on that (see the venv module).
Pyenv definitely solves it, but I talked about an officially supported solution.
Install them into different directories. There's nothing about python that makes this at all hard or even tricky.
https://github.com/jnwatson/py-lmdb/actions/runs/3457851983
It gets out of hand pretty fast.
Python 2 is now gone, but until it was, Nimpy was an easy way to write Python extension modules that only needed to be compiled once, and would work with any of your installed Python 2 and Python 3. Magic.
And testing document in a .sw_env_conf to keep the change vs the base.
If it is a re-runnable and readable doc we can trace it.
Sorry not solving the Abi but try to deal with the mess. Python is easy to learn and program. But running it is really. I document at least 7 Python version and control which one to use is itself hard.
I've seen plenty of packages with no source release.