pip, requirements.txt, and venv are madness. Look at how much suffering they cause! I was dealing with (different) pip issues today myself, and it's not an infrequent occurrence.
Rust's Cargo is a good reference point.
I wish you could just install all packages (in multiple versions!) into one place like /usr/lib/pythoncache, and the module loader would then decide which package version to load at import time. You could define your dependencies as usual in reqirements.txt or pyproject.yaml. And you could use either pip or the system package manager to manage the global package cache.
My last attempt at using pip this very morning to install a bunch of deps crashed and burn with an obscure python stack dump.
Oh, and, in your taxonomy of partial and inadequate solutions, you forgot to mention anaconda, possibly the most invasive package management solution I have ever seen, which exhibits the same problems pip does, i.e. where each install is a total crapshoot.
- Conda - Pipenv - Pip itself - Poetry
Which is quite symptomatic.
It’s odd how open source packages that big companies use can be reliant on so few people.
Edit: although checking the pip github, there more people committing so maybe I got something wrong, but there is far more activity starting around 2019.
Is there any good overview which packaging solution has which properties, and will work in specific situations?
Perhaps I'm just lucky but I never experienced any major issue with pip+venv. It always worked as advertised: create venv, activate venv, install/update pip locally, install packages listed in requirements.txt, and you're all set.
Which problems do you experience with pip?
If somebody else installs your package at a later time, pip will again install the latest fitting versions.
This install will be different from your install, or from other installs other people made in the meantime.
If a newer version if a dependency breaks your package because it has a backwards-incompatible change, your package won't work - and it will not show up in your testing because pip sees the existing, already installed packages and will think they are new enough. So, you'll find "it works for me".
A good examples are packages like python-opencv which break on python2 because the versioning implies they are backwards-compatible but they (or their given dependencies) use syntax which is not supported by python2.
And these things tend to snowball quickly because the number of packages which other packages depend on tends to grow exponentially, without a real upper bound.
And while there is lots of annoyment and cursing about package managers, I think a huge part of the problem is a cultural issue in the python community because people accept and create libraries with backwards-incompatible changes without marking that in the version numbers.
Obligatory link to Rich Hickey's brilliant talk on how to do it better: https://www.youtube.com/watch?v=oyLBGkS5ICk
And, as we are at talking about the future, how would you manage native extension modules which are written in Rust?
I've also collaborated and discussed with hundreds of other conda users, and except for speed and the lack of official lockfiles[0], there were no complaints, only praise.
[0] The functionality was always available, but not straightforward and not named as such.
Still, both poetry and pipenv work better for me than conda because they do have a lockfile mechanism which I find to be essential.