The problem is lack of leadership on this more than anything
So many things were fixed in python packaging.
- "-m" was introduced.
- Wheel replaced eggs.
- Manylinux target now exists.
- pypi was scaled uo, a new api
introduced and 2fa added.
- setup.py got replaced by
setup.cfg and now
pyproject.toml is gaining speed.
- the py launcher is a thing.
- Import priority changed.
- Import lib changed.
- Zipapps were added.
- Venv ships with python.
- Pip replaced easy_install,
then ensurepip was created.
- The new dependancy solver
was funded.
- Distutils has been sunsetted.
Setuptools is on its way.
A lot has been done, and the situation is 10 times better than it used to be. It is still bad, but not because nothing was done, simply because there is still a lot to do.
People dont realize the sheer scale of the task, and the inertia that comes with a language as successful and old as python.
You have a language that is 4 years older than java, highly dynamic, yet uses fortran/c/assembly code in very popular lib (scipy/numpy) accross all popular OS, and webassembly and arm, in 32 and 64 bits. People routinely install multiple versions of python on their machine, linux repos freeze the upgrades and split the packages, mac and windows stores ship broken versions, and half the python userbase is composed of people that are not professional coders that can't use a terminal. Of course companies will complain if you change anything that breaks their 5 years old server, and devs will complain if they don't get the last new great feature.
It's a really, really hard problem, dealt by a FOSS community that 10 years ago was still running on mostly volunteers and the budget of a small start up. Of course, just to get heat by comments on social media and no thanks for all the effort, as if you own total strangers your free work, with no flaws on top of that.
Not to mention you can imagine packaging is not the only things the core devs have to work on.
One would think that two decades of package improvement initiatives should have been a strong enough signal to PSF to prioritize building an official standard solution.
A third party can ignore legacy. It can break stuff. It can avoid supporting platform. It can have a big bug and push a fix the day later. It can skip red tapes. It can document after the fact. It can drop features the next year. It can requires pypi dependancies when the stdlib is not sufficient. It can be by a single author that don't have to ask anybody what they think about before creating something. It can skip security issues and focus on practicality.
CPython cannot.
So any change to the packaging story will always take years for the smalles thing.
That's the same for removing the GIL: touching the c api is a huge deal for the scientific and machine learning stack.
That's why requests has never been included in python despite being vendored with pip: you can never update it fast enough once it's in the stdlib.
That's why we don't have node_modules equivalent yet: autoloading code in the current directory would be a big security risk for a language that is included in many os by default so we need a good design.
Don't assume the team is incompetent or deaf. Given what has already been achieved, that would be a total mismatch.
I am not making any such assumptions (and I don't think most sane people are either); people are just unhappy with lack of an officially standard solution given that it was clear quite a while ago that it was needed. I also don't understand the comparison with GIL or C API - changes to them could be a breaking change whereas a fresh officially recommended solution would by definition be a new solution i.e. not a breaking change. It can be introduced and it can go through iterations while people can take their time to migrate from the myriad of third-party packages (or not, if they are happy with what they are using).
What people, who don't want to deal with deciding between or putting their faith in longevity of third-party tools, are looking for is a solution that is official, part of the standard library and is managed by PSF. That way the decision is basically made for them e.g. like it is done in Rust via cargo.
But the dirty secret of python packaging is that most of the problem don't come from packaging, but from boostrapping python. And this is a huge can of worms I didn't even mentioned.
Also, plenty of things require the participations of other communities like debian splitting pip out of the main package or anaconda not being compatible with pip.
All in all, the way you hand wave the problem is typical from the critics that have a very narrow view of the situation.
If it was that easy, it would have been done.
To be perfectly clear, I don't think that. I just think its a missed opportunity over the years to better ways of environment and package management. For instance, on the folder auto loading code: make it a feature of only virtual environments. I know python can detect that its in one. Or engage with distros not to build Python to allow this. They already do custom builds with it (its always missing something from a standard Python installation after all).
The first thing we need is a whole ne way of installing python, because half of the problems stems from that. That's just a huge endeavor.
Curiously, I wonder whether current leadership are unwilling to acknowledge that the current problems are essentially a leadership problem. See for example my short conversation with Pradyun on Mastodon (https://mas.to/@maegul/109726564552419983) where I’m not sure they were being open and rational (though they clearly know more than me).
This thread was in response to their blog post in Python packaging: https://pradyunsg.me/blog/2023/01/21/thoughts-on-python-pack...
Imo this isn't really viable because you will eventually run into version conflicts in the transitive dependencies of your the Python applications you're using/developing on your system, on most operating systems.
The version(s) that ships with an OS should only be used for shipping applications that are themselves part of the OS/distro.
The packaging solution is pip? (although for some reason it doesn't come with the now standard wheel, so it's less useful that it should be)
It's quite funny to see languages fumbling on dependency management and somehow "refusing" to look at how bundler operates, then stumbling on a very close solution and then community rejoices at their package management being the best thing since slice bread.
I did not know the bundler authors were actively contacted though, thanks for that piece of info!
You know, prior art that appeared about 5 years later and had huge adoption.
Insularity is a software ecosystem disease.
GroupIds are reverse DNS notation based on a subdomain you control. They both group dependencies (duh!) and avoid the top-level squatting problem. It's much harder to mix up http-utils like in npm and pypi, since there's no http-utils. There's com.google:http-utils and there's org.apache:http-utils (all the names are made up for example purposes).
Maven has a local centralized artifact repository/cache where artifacts are downloaded and then referred to by every project. In Java there isn't even a need for symlinks, since Java has a dynamic classpath aka the places where the libraries are searched. Though Python could do the same thing with the PYTHONPATH.
Maven is plugin based and it's reasonably easy to build your own, so it's been extended like crazy to do all sorts of wonderful and loony things like build C++. It's actually expected to be extended, the tool itself implements Java builds through plugins, the core doesn't do that.
Oh, from the start it had tools to cache/proxy your own packages. So you'd be able to have a mini-centralized repo for your company with just the stuff you need, in case the main repo is down. Artifactory, Nexus, there are others. And not just cache/mirror, but proxy: you'd call your local repo, and if it wouldn't find the thing, it would download it from internet repos you configure.
Another thing, the package format and repo structure are simple and straightforward. So Gradle, sbt, other fancier Java build tools just use Maven repos.
There are a million other things like that, it's a very robust tool with a very robust ecosystem.
The main pain point is the early-2000s style XML configuration format :-|
That makes a lot of people avoid it due to verbosity. But you know what? The format is stable. It's quite readable. There are a gazillion tools to manage it, IDEs have advanced autocompletion for it, etc. And I'm not saying that other folks should copy the config file format, just the rest of the ideas. Heck, even Maven has Polyglot Maven, to use different config file formats (not super adopted, but it's there).
I would much rather have binary redistribution left up to downstreams.