Hacker News Reader

Comment by koe123 | Hacker News Reader

Is my understanding correct that this would provide version agnostic python bindings? Currently, I am building a version of my bindings separately for each version (e.g. building and linking with python 3.7, 3.8, etc.). While automated, it still makes CI/CD take quite a long time.

View on HN

As others have said, this has been supported since the limited/stable APIs were introduced. What this adds is a way of implementing a Python extension that can be loaded in (not just compiled for, which is already an improvement!) different Python implementations, namely CPython, Pypy and GraalVM.

Cpython also has a limited stable abi and cp3X-abi3 wheels are compatible across multiple versions of Python.

https://docs.python.org/3/c-api/stable.html

But it is very limited. Understandably so, as they don't want to ossify the internal APIs, but it still is so limited that you can't actually build anything just using just that API as far as I know.

I checked now - polars is built using py abi3 wheels, and for me that means that you can build something substantial using the stable ABI! :)

See https://pypi.org/project/polars/1.9.0/#files cp38-abi3 wheels means they are compatible with cpython 3.8 or later.

Woah! Okay, that's very cool. I thought it was much more limited than that (for the stable abi). Awesome!

It seems like they use mostly normal python as a bridge with the rust codebase. So from what I've seen on their repo, they mostly do not use any CPython APIs (a part from a few wrappers I think). Which makes sense!

That makes sense: I would assume polars mostly converts from Python to rust at the edge then it works in rust internally.

Though I’ve not really looked at the details I’d assume most of the missing stuff would be “intimate” APIs of builtin types. And all the macros leveraging implementation details.

You can already build a single wheel as long as you only target cpython, if your needs fit with the limited / stable abi (abi3).

While pypy and graal have API support they don't have abi / abi3 support, so they still have to be built on their own (and per version I think).

While automated, it still makes CI/CD take quite a long time

See about using ccache -- https://ccache.dev/

I wouldn't recommend ccache (or sccache) in CI unless you really need it. They are not 100% reliable, and any time you save from caching will be more than lost debugging the weird failures you get when they go wrong.

please provide evidence for this assertion.

You can't cache based on the file contents alone. You will also need to cache based on all OS/compiler queries/variables/settings that the preprocessor depends on, since the header files might generate completely different content based on what ifdef gets triggered.

And that’s not impossible, just tedious. One tricky (and often unimportant) part is negative dependencies—when the build depends on the fact that a header or library cannot be found in a particular directory on a search path (which happens all the time, if you think about it). As far as I know, no compilers will cooperate with you on this, so build systems that try to get this right have to trace the compiler’s system calls to be sure (Tup does something like this) or completely control and hash absolutely everything that the compiler could possibly see (Nix and IIUC Bazel).

In C++ the __has_include preprocessor expression has been standardized since C++17, I'm not certain if C has standardized it yet though.

It’s not about that, that’s not relevant to ccache at all. (And yes, C23 does have __has_include, though not a lot of compilers have C23 yet.) It’s about having potentially conflicting headers in the source file’s directory, in your -I directories, and in your /usr/include directories.

Suppose a previous compile correctly resolved <libfoo.h> to /usr/include/libfoo.h, and that file remains unchanged, but since that time you’ve installed a private build of libfoo such that a new compile would instead resolve that to ~/.local/include/libfoo.h. What you want is to record not just that your compile opened /usr/include/libfoo.h (“positive dependencies” you get with -MD et al.), but that it tried $GITHOME/include/libfoo.h, ~/.local/include/libfoo.h, etc. before that and failed (“negative dependencies”), so that if any of those appear later you can force a recompile.

Oh yeah that can cause lots of weird problems. I've run into that sort of issue a lot when cross-compiling, cause often then you might have a system copy of a library and a different version for the target, that can be a real pain.

Maybe run every build version in their own container?

please read the documentation before dispensing uninformed advice like this -- it works using the output of the preprocessor and optionally, file paths

Why are you so skeptical? Think about how it works and then you'll understand that cache invalidation bugs are completely inevitable. Hell, cache invalidation is notoriously difficult to get right even when you aren't building it on top of a complex tool that was never designed for aggressive caching.

Just search the bugs for "hash":

https://github.com/ccache/ccache/issues?q=is%3Aissue+hash+is...

Here’s an example of a common Rust dependency which had a very subtle caching issue: https://github.com/dtolnay/thiserror/pull/279

I believe so, but it would presumably depend on what features you use.

Reply on news.ycombinator.com