I've started a series of blog posts talking about the Python packaging ecosystem and the many things that are going wrong with it. I know there are countless pieces out there already about the "mess" (that's in fact just about the first point I make); my goal with this series is to get into the details of why things are so awful, and uncover the underlying faults.
As a bit of background: I came here originally (https://news.ycombinator.com/item?id=41286179) because I saw that this was one of the relatively few places talking about Tim Peters' suspension from Python core development, which in turn was related to my own ban from the Python Discourse forum (https://discuss.python.org). While I joined that forum for other reasons, I found that most of my time there was spent on discussions of the packaging system. I used to not think about that very much - I was already able to make installable packages for simple, pure Python projects and have them work just fine, and I'd learned a few habits to avoid the most common pitfalls. But over the last year and a half or so, I've been making myself an expert on the topic.
In the first post, I go over a grab bag of common complaints that I don't think are major problems, or where people commonly seem to have unrealistic expectations. The Python world is fundamentally different from that of other "modern" languages with an "ecosystem": it's much more common to rely on compiled code in C or another similar language (and thus have to deal with an FFI, as well as figuring out who's going to build the code and when); the language is much older (it was designed in an era when a rich standard library was desirable because you couldn't rely on grabbing code from the Internet) and just plain works differently (the way imports work makes it unfeasible to support multiple versions of the same library in a given environment); etc.
This is of course just setting the stage for being able to rip into people for the things that really are inexcusable ;)
This piece ran about 4500 words, which is quite a bit more than I planned on (and I ended up splitting the writing across two days). Some of you might recognize things I've said in previous HN comments. For a bit of entertainment value, I've set the section headings to a musical theme.
Future planned topics for the series include, in no particular order:
* A "part 0" to go over basic concepts for people not already familiar with Python packaging
* A guide to a bunch of the tricks I use to mitigate these problems
* Security issues with Pip and its over-eagerness to build sdists (and thus run `setup.py` etc.), and the history of that problem
* The specific history of Setuptools taking over from Distutils
* The absurdly high count of Setuptools downloads from PyPI (https://pypistats.org/packages/setuptools)
* The awful metadata standards, and how legacy support for `setup.py` tricks has been holding things back
* Why wheels don't use better compression or internal symlinks, and the consequences for PyPI's bandwidth
* A bunch of stuff about the internals of Pip and Setuptools, and how awful they are (I'm not sure how this will be organized, but I especially want to cover the quasi-standard status they have despite not being part of the standard library)
* A discussion of why people don't seem to like venvs very much, and why they really aren't that bad (spoilers: a lot of the pain is really Pip's fault! And, very indirectly, Setuptools' fault too.)
* The social issues that are making it harder to develop standards (and get tool authors on board)
* An overview of the build backend I've been writing, `bbbb`
* A design for an application and library installer that I'd like to replace Pip and Pipx, which I'm calling Paper - the Python Application, Package and Environment wRangler