Why is Python packaging such a mess? That's a serious question. Has someone written up some recap or history piece on how it came to be this way? Any pointers?
(2) has historically also been a strength of Python packaging, and has been a big source of modernization efforts over the years that (IMO) have made packaging much better: things like pyproject.toml, build backends, etc.
(I think there have been a few recap talks at PyCon over the years on packaging's history. There are definitely people way more qualified than me to provide a more detailed summary :-)).
(If we use similarly-aged interpreted languages as a benchmark, I would say that Python packaging has modernized admirably in some aspects: compare CPAN, for example.)
- Minimal standardization of ways to cache pre-built artifacts for modules with binary/XS components (no widely adopted wheel equivalent).
- Extremely poor support in cpan-the-tool for multiple local::lib environments.
- Abject lack of standardization on how hermetic/isolated (or not) builds involving linking to native libraries/compiling code should be. Talking super simple stuff here: what path-related envs to inherit, when module build scripts should hardcode vs. use system-path-discoverable versions of common C toolchain components, stuff like that.
- Standardization on build tools ("backends" in Python parlance) is poor. Module::Build is "the future", but lots of stuff seems not to use it. ExtUtils::MakeMaker is ... not good, and alternatives seem to be all over the place in terms of supportedness (I've heard Dist::Zilla is good, never tried it; spent plenty of time with SWIG hassles though). This is a low bar, and Perl trips over it. To be fair, so does Python.
- No uninstallation support (because there's no standardized manifest format). Sure, cpanminus exists (mirroring the annoyance that is e.g. conda competing with pip in Python), and even then uninstallation is dicey.
- Uninstallation troubles hint at the larger issue: CPAN packages are, if you zoom out far enough, shell/perl scripts that perform arbitrary system modifications. As jsiracusa used to say when we worked together, "they're file sprayers, not packages". If you want anything more predictable (or less dangerous) than that, you're out of luck unless you follow a very narrow path which many packages and deployment environments may not be compatible with.
- TMTOWTDI doesn't help here; even simple pure-perl modules have piss poor consistency re: how they install, so answering questions like "what will installation of Some::Module make available" is harder than it needs to be because maintainers are by turns over-clever and using dated methodologies for packaging their code.
- For reproducibility, cpan-the-tool's stance on lockfiles and hashes seems to be "what the fuck is a lockfile?". Given poor CPAN-wide compliance with Semver or equivalent conventions, that makes builds that need to happen in different environments (e.g. local, docker, CI, production) needlessly complicated.
- MacOS support for many common libraries with XS/native components is very poor even by Python's (nightmarishly bad until c. 2019, when tools matured even though the advent of ARM macs made it feel like they didn't) standard.
I sincerely hope that stuff has improved in the Perl community since I last tangled with CPAN issues on the regular (around 5.18/5.20). But given the increasing number of lonely tumbleweeds drifting around CPAN and the Perl community's tendencies in general, I won't hold my breath.
Indeed it is basically incompatible with SemVer given that SemVer has its own way of marking unstable versions (or, equivalently, pre-release versions; SemVer conflates the two concepts). Unstable/pre-release versions under SemVer are indicated by a hyphenated suffix after the patch version - e.g. 1.2.3-alpha
Perl has the best packaging community I've seen. Ironically I don't think they even have binary packages, but they do have a system that tests both core Perl and most of the CPAN archive on like 20+ platforms. The packaging fits the community, it's robust, and sets conventions that the rest of the community follows that improves the entire ecosystem. It has definitely evolved over time. But I don't know of anyone who has ever said "Man, Perl packaging is a huge pain in the butt, I wish someone could solve this". If there's a problem, usually there is just one solution that everyone adopts, builds on and improves (which is ironic considering Perl's motto)
Whereas with Python, if there's a problem, someone tries to solve it, but it's not perfect, so somebody makes a second solution, then a third, fourth, fifth, etc. Python never established the convention of reusing and improving existing things. Python has a forking/independent culture. So there will always be 20 different ways to do something and none of them all that great (which is ironic considering Python's principles)
This was the write up I did last year
I have no beef with Poetry but having taken huge brain damage dealing with Python packaging as a USP for end users, the maintainers have decided to not participate in the most exciting things in the ecosystem to the detriment of the otherwise excellent and thoughtfully designed project. Ultimately people who are offering a packaging alternative must deliver a “universal install wizard” solution that includes Windows, which interests few people to work on.
That avoids the problems you can get where everything gets discussed to death and nothing actually gets written, but the downside was that the project spent a long time in an uneasy situation where it didn't have complete official buy-in but was popular enough that it would have been hard to push an alternative.
There were a couple of questionable early design decisions that didn't help.