Comment by woodruffw | Hacker News Reader

Comment by woodruffw | Hacker News Reader

Parent Full thread

> I'm going to admit that what I really want to see is a strong push to standardize and fully incorporate package management and distribution into python's core. Despite the work done on it, it's still a mess as far as I can see, and there is no single source of truth (that I know of) on how to do it.

Package management is standardized in a series of PEPs[1]. Some of those PEPs are living documents that have versions maintained under the PyPA packaging specifications[2].

The Python Packaging User Guide[3] is, for most things, the canonical reference for how to do package distribution in Python. It's also maintained by the PyPA.

(I happen to agree, even with all of this, that Python packaging is a bit of a mess. But it's a much better defined mess than it was even 5 years ago, and initiatives to bring packaging into the core need to address ~20 years of packaging debt.)

[1]: https://peps.python.org/topic/packaging/

[2]: https://packaging.python.org/en/latest/specifications/index....

[3]: https://packaging.python.org/en/latest/flow/

which is why I said, "Despite the work done on it"

Yes, that was meant more for the “source of truth” part.

No, I do appreciate you taking the time to do it (I was too lazy)

Is there anything in there about managing dependencies within a python project? What is the canonical way to do that in python today?

It depends (unfortunately) on what you mean by a Python project:

* If you mean a thing that's ultimately meant to be `pip` installable, then you should use `pyproject.toml` with PEP 518 standard metadata. That includes the dependencies for your project; the PyPUG linked above should have an example of that.

* If you mean a thing that's meant to be deployed with a bunch of Python dependencies, then `requirements.txt` is probably still your best bet.

I meant the second. requirements.txt is a really bad solution for that, and that is the frustration many of us have that have used languages with much better solutions.

Requirements feels like a dirty hack but it does work fine. It has ==, ~=, and >= for version numbers, as well as allowing you to flag dependencies for different target os, etc. And then you can add setup.py if you need custom steps. But yes, it feels dirty to maintain requirements.txt, requirements-dev.txt, etc.

Poetry is the most common solution that I've seen in the wild. You spec everything using pyproject.toml and then "poetry install" and it will manage a venv of its own. But you still need to tell people to "pip install poetry" as step 0 which is annoying.

If you don't care about deploying python files, and rather just the final product, I'd recommend either nuitka or pyinstaller. These are for bundling your project into an executable without a python runtime needed (--onefile type of options for single file output). Neither supports cross compilation though.

What flow do you use with requirements.txt that gives you reproducible builds across a team and environments? Using ==, ~=, and >= will not give you reproducible builds.

Hash-pinning with requirements.txt will get you the closest to this, but it's not possible in the general case to have a cross-environment reproducible build with Python. The closest you can hope for is a build that reproduces in the same environment.

This problem is shared by the majority of language packaging ecosystems; the only one I'm aware of that might avoid it is Java.

How do you get a reproducible build in python for the same os/arch? As in, what concrete steps do you take?

This is very easy in nearly every other language that is popular. No one ever answers this clearly in threads like this short of saying “use poetry” which makes my point. I’ve asked many times.

I explicitly said that you can't. Python's packaging ecosystem wasn't designed with reproducibility in mind, and has never claimed to prioritize reproducibility. The best you can do is get close, and hash-pinning gets you pretty close.

I'm not aware of any other major language or language packaging ecosystem that makes reproducibility straightforward. Certainly not Ruby or NPM, and not even brand new ones like Rust's Crates. Java appears to be the closest[1], but is operating with significant advantages (distributing reproducible bytecode to all users, minimizing system dependencies, etc.).

Edit: In addition to hash-pinning, you can instruct `pip` to only install built distributions, i.e. wheels. If you do both hash-pinning and built distributions only, your package installation step _should_ reproduce exactly on machines of the same OS, architecture, and Python version. But again, this is guaranteed nowhere.

[1]: https://reproducible-builds.org/docs/jvm/

You are splitting pedantic hairs in order to avoid talking about the obvious. Python's dependency management is much worse than ruby's and nearly every other popular language.

  In ruby you add dependencies to a Gemfile then ...
  $ bundle install
  $ git add Gemfile Gemfile.lock

and other members of your team can have the same build as you.

requirements.txt doesn't solve this basic need.

Edit: formatting.

I think you're confusing lockfiles with reproducibility. Lockfiles are good, but they don't guarantee reproducibility: a locked (or pinned, hashed, etc.) dependency might always be the exact same source artifact, but it can install in different ways (e.g. due to local toolchain differences, different versions of dependencies, sensitivity to timestamps, sensitivity to user-controlled environment variables, etc.).

Reproducibility is a much harder problem than dependency locking, and (again) I'm not aware of any language level packaging ecosystem that really supports it out of the box.

Python doesn't have reproducible builds, but it does have lockfiles (via hashed and pinned requirements). They're not particularly good (for all the reasons mentioned upthread), but they do indeed exist. If you use them as I've said, then your environment will be approximately as repeatable as with any other language packaging ecosystem (and arguably more so in some cases, since wheel installs are reproducible where gem installs aren't).

What is the most common python flow for dependency locking? Is it in any of the PEPs?

That would be PEP 508.

508 notably excludes hash-pinning, which is a significant limitation -- hash-pinning is defined by how `pip` implements it.

Thanks, is there any kind of tutorial on how you should use this in a python project?

Edit: I think this is one of the biggest problems someone coming to python has. Python advocates say some version of, "you can roughly do that" but there isn't a clear explanation of how to do it.

Edit 2: I see that the official docs have a Pipenv flow outlined. Is Pipenv the way people do this in python these days?

> Thanks, is there any kind of tutorial on how you should use this in a python project?

The PyPUG docs contain examples of using all of the PEPs mentioned above. TL;DR: all you need for 99% of use cases is a pyproject.toml.

Those docs say to use Pipenv, or am I looking at the wrong docs? Really not sure why python people can’t articulate a clear flow to follow. It’s all riddles.

Binary wheels are just archives, I don't see differences in the way they install between different systems.

Source wheels that contain C/C++ code are so annoying to install on Windows that we don't use them. But most packages provide binary wheels anyway.

> Binary wheels are just archives, I don't see differences in the way they install between different systems.

The subtlety here is in which binary wheel is selected: a particular (host, arch, libc) tuple may cause `pip` to select a more specific wheel for the same version of the package, or even an entirely different wheel. This makes wheels themselves reproducible between systems, but it also means that which wheel isn't guaranteed.

We get (very) close to cross-environment reproducible builds for Python with https://github.com/pantsbuild/pex (via Pants). For instance, we build Linux x86-64 artifacts that run on AWS Lambda, and can build them natively on ARM macOS. (As pointed out elsewhere, wheels are an important part of this.)

This is not raw requirements.txt, but isn’t too far off: Pants/PEX can consume one to produce a hash-pinned lock file.

Rust and Go both have proper lock files... Both of which are good enough to satisfy Nix's requirements for reproducability such that they re-use the lock file hashes. This "it's hard, no one does it right" feels like a cope.

We do it like this:

- configure the project with pyproject.toml,

- use pip-compile (from the pip-tools package) to create a lockfile,

- commit the lockfile into git,

- whenever we want to update the dependencies, do it through pip-compile again (if you give it an existing lockfile as output, it will keep what's in there and change only what's required).

Since all our requirements are cross-platform and on PyPI, we can install the same env everywhere.

> * If you mean a thing that's meant to be deployed with a bunch of Python dependencies, then `requirements.txt` is probably still your best bet.

This is exactly how we got in this mess. Using ``setup.cfg`` or ``pyproject.toml`` for all projects makes this easy as now your deployable project can be installed via pip like every other one.

1. ``python -m virtualenv .``

2. ``source ./bin/activate.fish``

3. ``pip install -U https://my.program.com/path/to/tarball.tar.xz``

Wait, why are ghey two different things? Can't pip be used for deployment?

It can -- the second flow also uses `pip`.

The terminology here is confusing: the first is the flow that produces a "distribution" (i.e., an sdist or bdist), while the second is the flow that produces an "environment" (i.e., a specific set of packages installed in some prefix).

Reply on news.ycombinator.com