Hacker News Reader

Comment by woodruffw | Hacker News Reader

This is a great writeup by a central figure in Python packaging, and gets to the core of one of Python packaging's biggest strengths (and weaknesses): the PyPA is primarily an "open tent," with mostly independent (but somewhat standards-driven) development within it.

Pradyun's point about unnecessary competition rings especially true to me, and points to (IMO) a hard reality about where the ecosystem needs to go: at some point, there need to be some prescriptions about the one good tool to use for 99.9% of use cases, and with that will probably come some hurt feelings and disregarded technical opinions (including possibly mine!). But that's what needs to happen in order to produce a uniform tooling environment and UX.

View on HN

To me the natural choice would be pip which, as OP points out, has the advantage of shipping with Python by default. We could remedy the shortcomings of pip by incorporating the most successful features from the N alternatives that exist today.

The npm/yarn fork that formed in 2016 is the closest analogy I can think of. My impression is that npm improved on its biggest weaknesses (e.g. a lock file) and has remained the default choice in that ecosystem. I suspect the same would happen with pip if it too made significant improvements.

Shipping with python is a disadvantage; any real workflow tool needs to be able to manage multiple python installations, and if you try to use a part of the python installation to do that you've got a circular dependency problem. Plus the whole "where modules go to die" thing.

Does anything speak against flipping the relationship? i.e. using pip (or something else) as the standard way to install Python.

pip is a python program.

You might want to look at micromamba as a separate tool that can install python itself, libraries, etc. with zero dependency on a python being installed. It's a very fast tool written in C++ so it's easy to install and use almost anywhere. It uses the conda ecosystem of packages.

If you have multiple versions of python installed you can just do

python3.9 -m pip

python3.10 -m pip

The standard library is the reason why python is popular to begin with.

To me, that’d be a UX nightmare. Even if I tried to, I’m not going to memorize which of my 20 checked-out Python projects are requiring which Python invocation.

IMHO it’s absolutely the toolchain’s job to manage that for me. I’d never adopt a toolchain that wouldn’t.

You do know that python is backwards compatible right?

You can just use 3.11 with everything.

Traditionally it wasn't, and even a "minor" upgrade was a significant undertaking for a large project. Maybe that's changed.

Maybe in the 2.4 -> 2.7 transitions - Python3 after about 3.6 is pretty good, you might see some minor package breakage if you're running the very latest version, but that's fairly rare.

Generally speaking, if you're on a modern OS - like recent Ubuntu LTS or MacOS you're getting 3.8 -> 3.10 and compatibility is very good in these releases.

When exactly are you defining as “traditional”? The Python point releases have generally been painless for a long time. There was a little friction in the early 2-3 era before they made it easier to write code which worked on both without translation but in general it’s been at least a decade since I’ve cared about this except for one time where a couple of libraries depended on a private API which changed in Python 2.7.9, and that was back in 2014.

https://bugs.python.org/issue22438

If my project is a library that needs to support e.g. 3.8 through 3.11, then I’m not going to use 3.11 for my development environment.

The same goes if my project depends on a library or framework that doesn’t support 3.11.

To some extend. NPM is still (even after all the improvements) the weakest of the package managers.

There’s no doubt it’s the default though.

Even if pip was a little better I still wouldn’t use it, because Poetry solves the problem so well you’d need to pull it out of my cold hands. It solves not only the install package problem, but locking, venvs, project structure and packaging in a well integrated solution with a fantastic UX.

If poetry (or similar) were renamed to Pip and it were to be included in the stdlib id obviously switch.

It doesn’t solve the single static binary problem or docs problems, but it’s much much closer.

Everytime you install a package through poetry, you're also running pip code.

You didn't even understand the point they made: the need for Pipenv and poetry would pretty much go away if pip added support for a proper lockfile and venvs. And that's the only correct choices as pip is already pythons package manager.

What's the problem with first doing the venv thing and then running pip?

Any multi step process is guaranteed to produce random assortments of workflow tools to manage those steps which will seek to replace the original process as the one true process.

If there two steps, one part of the community will insist they belong in: a makefile, bash script, python script, lambda network service, bazel, pants, scons, terraform, ansible, npm …

So we should have 1 single shell command to do literally everything?

before poetry and pipenv was a thing, pretty much everyone was recommending the virtualenvwrapper script instead of using venv directly... so yes, thats what it ultimetaly always ends up as.

One might invest 20 seconds to learn how to use it, but to each their own :)

Not what I would or wouldn’t do personally, but what happens generally in practice.

Poetry ended up being mostly a disappointment having used it for the last 18 months. First time installs were eventually an hour long to resolve, and lock files were platform specific, making using them for reproducibility almost useless. The wider python ecosystem is frankly painful to productionize even today and it really does need some focus and governance.

Are you sure the lockfiles are platform-specific?

I spent a long time migrating a project to use poetry. One of the reasons I opted for poetry over others was that the lockfile retained all of the environment markers in the packaging metadata, so that the lockfile could support multiple interpreters and interpreter versions, multiple platforms, etc.

I can't speak for Poetry directly, but knowing how Python dependency resolution works: I don't think Poetry can make lockfiles not platform-specific, since package source distributions are allowed to (and regularly do) run platform-specific code for their own dependency selection logic.

For example, your package might depend on `foo`, which in turn could sniff the host OS and select the appropriate subdependency. You'd then end up pinning that subdependency, which would be incorrect on a different host OS.

(Similarly for Python versions: a subdependency might be required on < 3.7, so re-installing from a lockfile generated from an older Python could produce a spurious runtime dependency.)

If you use C-extensions or platform specific packages like pywin32, how would.you expect them to NOT be platform specific? Would you want it to only allow pure Python packages?

I've given up on poetry. I still to this day can't get it to use the active version of python, which you'd think would be the default. Even after setting the config flag. Setting the config flag somehow made things worse, all my poetry envs started using the wrong python. I can't even force it to use the version I want with 'env use /path/to/python', I can only pass stuff like 'env use 3.9'. What the hell is 3.9? I want to pass a specific path to a version I have.

> Pradyun's point about unnecessary competition rings especially true to me, and points to (IMO) a hard reality about where the ecosystem needs to go: at some point, there need to be some prescriptions about the one good tool to use for 99.9% of use cases, and with that will probably come some hurt feelings and disregarded technical opinions (including possibly mine!). But that's what needs to happen in order to produce a uniform tooling environment and UX.

Not to mention that all efforts would be focused on improving a single package manager. Even if the one that was standardized wasn't the "best" at first (by whichever metric you prefer), giving everyone an incentive to improve that one will likely make it the best within a few years.

> some prescriptions about the one good tool to use for 99.9% of use cases

And we've always had such tool, the humble "pip". Now it even works great with pyproject.toml. Plus built-in "virtulalenv" if you need isolation.

The primary (probably build and) packaging system for a software application should probably support the maximum level of metadata sufficient for downstream repackaging tools.

Metadata for the ultimate software package should probably include a sufficient number of attributes in its declarative manifest:

Package namespace and name,

Per-file paths and Checksums, and at least one cryptographic signature from the original publisher. Whether the server has signed what was uploaded is irrelevant if it and the files within don't match a publisher signature at upload time?

And then there's the permissions metadata, the ACLs and context labels to support any or all of: SELinux, AppArmor, Flatpak, OpenSnitch, etc.. Neither Python packages nor conda packages nor RPM support specifying permissions and capabilities necessary for operation of downstream packages.

You can change the resolver, but the package metadata would need to include sufficient data elements for Python packaging to be the ideal uni-language package manager imho

Reply on news.ycombinator.com