Python Packaging, One Year Later: A Look Back at 2023 in Python Packaging
chriswarrick.com
chriswarrick.com
I don't buy this complaint (?) about Ruff. The broader point is that Python is _so good_ that people are willing to learn Rust to make the Python experience even better.
I use setup.py legacy because of my semi-complicated requirements for C and Cython-based extensions (including a code generation step, and per-file compilation options, and compiler-based compilation and link flags). Knowing that SciPy and NumPy managed to migrate means Meson might be the right option for me.
The PyPA docs about "Packaging binary extensions" at https://packaging.python.org/en/latest/guides/packaging-bina... still says "Last Reviewed 2013-12-08".
It's interesting that the original post (https://chriswarrick.com/blog/2023/01/15/how-to-improve-pyth...) complained about there being too many options (whilst missing the "scientific python" and rust backends, and including wheel (which is part of setuptools now) and venv (which is primarily a library)), when much of the original push to standardise packaging came from the scientific python community (see e.g. https://pypackaging-native.github.io/), and the need to get away from a distutils/setuptools monoculture (given previous attempts at improving the situation e.g. bento).
... oh, wait! TIL "The easiest way to get CMake is to add it to the pyproject.toml file. With pip 10 or later, this will cause the CMake Python package to be downloaded and installed when your project is built."
99 of the backends use scikit-build-core.
Only 11 of the backends analyzed used enscons.
There are too many options. "There should be one-- and preferably only one --obvious way to do it." is at odds with the "let a hundred flowers bloom" of the current build backend philosophy.
The obvious way to do it is to start with the Python documentation.
https://docs.python.org/3/extending/extending.html only mentions modifying Modules/Setup.local from a Python source distribution(!).
https://docs.python.org/3/extending/building.html#building says:
"Python 3.12 and newer no longer come with distutils. Please refer to the setuptools documentation at https://setuptools.readthedocs.io/en/latest/setuptools.html"
My code works with setuptools because I started this project 15 years ago, and it's been incremental changes to my setup.py ever since. I find it hard to figure out which option I should use, because so few people use these alternatives and there's very little guidance - either direct advice from PyPA or hits from DDG - about how to handle the issues I've already resolved in setuptools.
Pypackaging-native even describes the special things I need, and which I've solved with setuptools:
- how do I have per-file compilation options so I can optimize for different architectures? These are also compiler specific.
- how do I enable/disable compile-time support for OpenMP? And specify the right link flags? And provide binary wheels for macOS with OpenMP?
- how do I add a code generation step?
- how do I have the source distribution include the Cython-generated C source so my clients don't need to have Cython installed? (My experience is there's about a 20% chance my customers have an incompatible Cython installed in their environment. Distributing C source means they can build without issues.)
The fact that NumPy successfully switched to Meson tells me Meson will likely handle what I want, I just have to figure how to do it. Which I do not look forward to.
That I now know CMake is another option makes the paradox of choice rear its ugly head.
Thing is, my setup.py is full of hacks and is a nuisance to use, so I would prefer a better solution.
What kind of hacks?
In the "python setup.py install" days I processed sys.argv to handle extra compile-time parameters, like "don't compile for OpenMP". When direct use of setup.py became unsupported, I switched to environment variables to pass in the configuration.
While it works, I don't know if it's a long-term viable solution. I read about the push towards isolated environments and reproducible builds, which is not something you can get if your build depends on ephemeral environment variables which are not captured in the pyproject.toml or dependency hashes.
(Remember, I distribute a source package, not a precompiled wheel.)
The longest component in my distribution takes a minute to compile. Setuptools does not handle dependency tracking well, so even if I touch C code which only takes a second to compile, it still rebuilds the entire component.
I have a hacked-up Makefile for when I really need to work on that part of the code and want faster turn-arounds, but I would prefer to have a unified build system that really understands build dependencies.
I subclass build_ext to handle the per-file compilation args, per-compiler link args, and to add an '$ORIGIN' rpath for non-Darwin builds as I generate shared libraries which depend on other shared libraries. To make this work I had to monkeypatch setuptools implementation internals.
I know how fragile it is, and would like something better, which I could depend on being around for another 15 years.
But when I dip into the Packaging world, I feel like my needs are in a forgotten corner, with the main impetuous being the needs of enterprise systems with engineering staff - a far cry from the packaging world I started with 20 years ago where building C extensions was one of the primary concerns.
I’m not optimistic for “Python Packaging Two Years Later” in 2025.