Is there anything in there about managing dependencies within a python project? What is the canonical way to do that in python today?
* If you mean a thing that's ultimately meant to be `pip` installable, then you should use `pyproject.toml` with PEP 518 standard metadata. That includes the dependencies for your project; the PyPUG linked above should have an example of that.
* If you mean a thing that's meant to be deployed with a bunch of Python dependencies, then `requirements.txt` is probably still your best bet.
Poetry is the most common solution that I've seen in the wild. You spec everything using pyproject.toml and then "poetry install" and it will manage a venv of its own. But you still need to tell people to "pip install poetry" as step 0 which is annoying.
If you don't care about deploying python files, and rather just the final product, I'd recommend either nuitka or pyinstaller. These are for bundling your project into an executable without a python runtime needed (--onefile type of options for single file output). Neither supports cross compilation though.
This problem is shared by the majority of language packaging ecosystems; the only one I'm aware of that might avoid it is Java.
This is very easy in nearly every other language that is popular. No one ever answers this clearly in threads like this short of saying “use poetry” which makes my point. I’ve asked many times.
I'm not aware of any other major language or language packaging ecosystem that makes reproducibility straightforward. Certainly not Ruby or NPM, and not even brand new ones like Rust's Crates. Java appears to be the closest[1], but is operating with significant advantages (distributing reproducible bytecode to all users, minimizing system dependencies, etc.).
Edit: In addition to hash-pinning, you can instruct `pip` to only install built distributions, i.e. wheels. If you do both hash-pinning and built distributions only, your package installation step _should_ reproduce exactly on machines of the same OS, architecture, and Python version. But again, this is guaranteed nowhere.
In ruby you add dependencies to a Gemfile then ...
$ bundle install
$ git add Gemfile Gemfile.lock
requirements.txt doesn't solve this basic need.
Edit: formatting.
Reproducibility is a much harder problem than dependency locking, and (again) I'm not aware of any language level packaging ecosystem that really supports it out of the box.
Python doesn't have reproducible builds, but it does have lockfiles (via hashed and pinned requirements). They're not particularly good (for all the reasons mentioned upthread), but they do indeed exist. If you use them as I've said, then your environment will be approximately as repeatable as with any other language packaging ecosystem (and arguably more so in some cases, since wheel installs are reproducible where gem installs aren't).
508 notably excludes hash-pinning, which is a significant limitation -- hash-pinning is defined by how `pip` implements it.
Edit: I think this is one of the biggest problems someone coming to python has. Python advocates say some version of, "you can roughly do that" but there isn't a clear explanation of how to do it.
Edit 2: I see that the official docs have a Pipenv flow outlined. Is Pipenv the way people do this in python these days?
The PyPUG docs contain examples of using all of the PEPs mentioned above. TL;DR: all you need for 99% of use cases is a pyproject.toml.
Source wheels that contain C/C++ code are so annoying to install on Windows that we don't use them. But most packages provide binary wheels anyway.
The subtlety here is in which binary wheel is selected: a particular (host, arch, libc) tuple may cause `pip` to select a more specific wheel for the same version of the package, or even an entirely different wheel. This makes wheels themselves reproducible between systems, but it also means that which wheel isn't guaranteed.
This is not raw requirements.txt, but isn’t too far off: Pants/PEX can consume one to produce a hash-pinned lock file.
- configure the project with pyproject.toml,
- use pip-compile (from the pip-tools package) to create a lockfile,
- commit the lockfile into git,
- whenever we want to update the dependencies, do it through pip-compile again (if you give it an existing lockfile as output, it will keep what's in there and change only what's required).
Since all our requirements are cross-platform and on PyPI, we can install the same env everywhere.
This is exactly how we got in this mess. Using ``setup.cfg`` or ``pyproject.toml`` for all projects makes this easy as now your deployable project can be installed via pip like every other one.
1. ``python -m virtualenv .``
2. ``source ./bin/activate.fish``
3. ``pip install -U https://my.program.com/path/to/tarball.tar.xz``
The terminology here is confusing: the first is the flow that produces a "distribution" (i.e., an sdist or bdist), while the second is the flow that produces an "environment" (i.e., a specific set of packages installed in some prefix).