Comment by geofft | Hacker News Reader

Comment by geofft | Hacker News Reader

This is a very weird post to read, as someone who does (occasional) distro Python work - I feel like Python absolutely is listening to us! It's just that it's hard work, and distro packaging is mostly done by volunteers, and there are a whole lot of things to work through.

Here's some work I and others did earlier this year, which I thought was a great example of folks from the core Python packaging world and folks from distros working together: https://www.python.org/dev/peps/pep-0668/ See the massive table of use cases for all the things we had to think about.

You might note that one of the things it needs is additional participation on the Discourse thread from the authors (like myself) and from other distros. Again. It's mostly work done by volunteers, and there's a lot to work through. There's no magic to it.

I can tell you that the following (from TFA) will absolutely make things worse for distros, though:

> I call on the PSF to sit down for some serious, sober engineering work to fix this problem. Draw up a list of the use-cases you need to support, pick the most promising initiative, and put in the hours to make it work properly, today and tomorrow. Design something you can stick with and make stable for the next 30 years. If you have to break some hearts, fine. [...] These PEPs are designed to tolerate the proliferation of build systems, which is exactly what needs to stop. Python ought to stop trying to avoid hurting anyone’s feelings and pick one.

If you want the PSF to fund some engineering work on its own that it can finish much faster than any volunteer packager can even read the proposal, break some hearts, stop proliferating build systems, and hurt people's feelings, they will absolutely do that and say "Distro packaging is not supported, we only support virtualenvs. Users should make virtualenvs. Distro software should ship virtualenvs. Installing a Python package systemwide is meaningless." That's clearly the best-supported option right now, and it's a surprisingly technically defensible answer, but it's not going to make you happy.

"use virtualenvs and install your dependencies from an unfiltered, unsupervised, untrusted source" is certainly the solution most aligned with the rest of the industry but I struggle to see any technical benefit from it. Other than perhaps "move faster and break more things".

Virtualenvs don't require using unfiltered/unsupervised/untrusted sources. They're a place to install things into, not a place to get things from.

The specific model that distros could adopt, if we went this route, is that each Python package builds into a .whl, they're build-dependencies of applications, and applications install .whls into a virtualenv at build time. You'd still restrict packages to come from the distro with the usual policies (built from source code, compliant with licensing, not contacting the network at build time, etc. etc.).

So, for instance, installing "python-somelib" would get you a /usr/share/python-wheels/somelib-1.0.whl, built from source. The build process of "someapp" would create a /usr/share/someapp/venv, pip install that wheel into that virtualenv, and then symlink /usr/bin/someapp to /usr/share/someapp/venv/bin/someapp.

The PEP goes into more details about why virtualenvs are recommended, and I can give you a whole host of subtle reasons, but that's not really the point - the point is that it's defensible, not that it's perfect, and that it's very easy to implement with what works today. So if you ask the PSF to come up with something that magically solves the problems and makes people sad if necessary, this is literally what they're going to come up with. If you don't like that outcome (and there are good reasons not to like it!), then you shouldn't ask for them to arbitrarily pick an outcome some people won't like.

The problem here is that distros only supply one version of the package which means you gain nothing by having that same version installed in multiple virtualenvs.

So I guess the problem actually lies in the python library ecosystem that's becoming a npm like dependency hell.

The relevant question here is probably whether the library ecosystem is like that because the packaging tooling sucks or is it the other way around?

Well, that's not true:

- Almost all distros don't just supply one version of a package. They try to avoid it, but I can't recall one with a hard rule against it. For instance, Debian packages multiple versions of autoconf https://packages.debian.org/search?keywords=autoconf2 , the Linux kernel, etc.

- One reason that distros try not to install multiple versions of a package is that it's hard to specify which one you want. If you have, say, requests 1.0 and 2.0 installed, which one does "import requests" get you? The virtualenv approach, where "import requests" simply does not work in an un-virtualenv'd Python, avoids this problem entirely. Each application can independently build-depend on python-requests-1 or python-requests-2, and each user can create their own virtualenv and pip install /usr/share/wheels/requests-1.whl or requests-2.whl as they prefer.

- Even if you do not have different versions of the same library, it may well be the case that you have two different libraries with the same importable name. As a great example, see CJ Wright's talk from PackagingCon last week "Will the Real Slugify Please Stand Up" https://pretalx.com/packagingcon-2021/talk/P3983F/ (I expect they'll post videos online soon). tl;dr there are three packages you can get via "import slugify", and they expose different APIs.

- The possibility you haven't accounted for is "The library ecosystem is like that because things are fast-moving, because people have actual problems they want to solve, and upgrading dependencies and sorting out conflicts is work." It would be great for that work to be done, but we go back to the problem I mentioned at the top - limited volunteer time. In the absence of time to engineer things perfectly, your options are to ship something that's engineered imperfectly or decide not to ship it. We went through the dark ages of "We'll ship the next Debian release when all the bugs are solved" over a decade ago, and it turns out that this doesn't actually help users in any way.

> The possibility you haven't accounted for is "The library ecosystem is like that because things are fast-moving, because people have actual problems they want to solve, and upgrading dependencies and sorting out conflicts is work."

Sorry, I was overly aggressive in my previous comments. All I want is the Python ecosystem to acknowledge some people prefer stable over fast moving, recognize the importance of that and work towards a solution that isn't horrible for their use case.

We all depend on someone somewhere deep in our stack caring for stability, even when we don't realize it.

They have acknowledged it, noted the use case, and decided that they are unable to support it. Python has this in common with most other programming languages.

Providing long-term backwards compatibility with the ability to share libraries and update them system-wise is difficult and imposes very high costs on an ecosystem. Only a small number of tech stacks have ever done this (for example, C and Perl) - and not only are they so old that it wasn’t a conscious choice, they have stagnated as a result.

Reply on news.ycombinator.com