Python lock files have officially been standardized | Hacker News Reader

Python lock files have officially been standardized

discuss.python.org

36 pointsofek17 comments

PEP 751: https://peps.python.org/pep-0751/

This is great. pip-tools` is so valuable right now in helping mitigate these dependency tree issues. I'd love to see some form of support in core Python. I really hope this becomes pushed out, similar to how `pip` actually has a dependency resolver now. Relying on running `pip freeze` to create a quasi-lock is a horrible pattern for enterprise environments and for packages. I'm really looking forward to how this turns out, even though it's still in a proposal-type phase. `

Pip has had a dependency resolver since 2020, it just had some issues initially (in large part because of the terrible state of metadata in the ecosystem). There are plenty of other problems with Pip, of course, and not having an obvious way to save and reproduce a package resolution is one of them.

Pip doesn't aim to manage packages in the first place, however; expecting to lock the entire environment is out of its wheelhouse (or should I say .whl-house?).

For Paper, rather than implementing anything like Pip's "dry-run" option (which is nowhere near as "dry" as you'd expect, but that's a separate issue...), I'm planning to have a separate command to output PEP 751 lockfiles representing an individual resolution, i.e. the packages that would be added to the environment.

What does this mean for pip-tools' requirements.in, Pipfile.lock, pip constraints.txt, Poetry.lock, pyroject.toml, and uv.lock?

I think the plan is to replace all of those.

The PEP has buy in from all the major tools.

It doesn't mention them in alphabetical order.

From last week re: GitHub Dependabot and conda/mamba/micromamba/pixi support:

"github dependabot, meta.yaml, environment.yml, conda-lock.yaml, pixi.lock" https://github.com/regro/cf-scripts/issues/3920#issuecomment... https://github.com/dependabot/dependabot-core/issues/2227#is... incl. links to the source of dependabot

It does, and that is the plan as far as I can tell. The prior discussion in the thread, as well as the commit history for the PEP (https://github.com/python/peps/commits/main/peps/pep-0751.rs...), shows pretty clear intent to take all of their needs into consideration.

During the, very long, discussion it became apparent that different tools have different design requirements on a lock file. So it's not really clear what will happen to other lock files.

It may be this new lock file acts as an interchange format, that all tools can consume or produce, but not something they internally use.

Though, maybe we're lucky and tools will be able to use it directly, or we might have to wait for a new version of the standard once tools have been able to work with it long enough to know the deficiencies.

Non-lock files, like pyroject.toml and requirements.in remain unaffected.

pyproject.toml is still fundamentally intended to be an input file that describes abstract dependencies for a single package. Lock files describe entire environments, or at least entire change-sets for an environment, and specify everything - pinned versions, including for transitive dependencies, and also hashes and other expectations for the install artifacts.

Tools that use custom lock files are at liberty to switch over, and any concerns about migration, backwards compatibility etc. are up to them. Several major tool authors were consulted for the design repeatedly across the discussion thread, over a period of months (and this is just the latest attempt at the design task; the total history is much longer).

uv has an issue up already to track implementation: https://github.com/astral-sh/uv/issues/12584

A more direct link (should also work for people viewing the Discourse forum in "infinite scroll" mode rather than with pagination): https://discuss.python.org/t/_/77293/150

Very personal nitpick, but I don't like prefixing things with py, it makes no sense to me. A lock is a lock, so why a pylock.toml and not a lock.toml? You do what you want with it. Same with pyproject.toml

Well in my project at work now I have a pyproject. toml and a dbtproject.toml, I'm sure there are more complicated projects out there

because python isn't the only language?

Python projects often need to share space with code written in other languages used for extensions - especially C traditionally, but now there are many competitors (and Numpy also uses Fortran for LAPACK and possibly also BLAS; I'd have to check).

PEP 518 gave the original definition for pyproject.toml contents, defining its use for describing build-time requirements. (PEP 517 appeared more or less in tandem, describing the hook to call an entry point for the build backend; later, PEP 621 standardized some metadata for the build backend to use.) Many other options were considered, including for the name: https://peps.python.org/pep-0518/#other-file-names

It's still a draft, so not officially standardized yet.

It's only marked as a draft because there has not been a PR to update the status as it was just approved this morning.

Reply on news.ycombinator.com