Hacker News Reader

Comment by epistasis | Hacker News Reader

I 100% agree on this. But it's been pretty hilarious on how obstinate tool makers are about changing to Python's new pyproject.toml [1]

Python packaging is a bit of a mess already, so when I recently started a new small package I wanted to choose tooling that would not clutter, but a fair amount of tool makers were reluctant to allow code into their repo that would use the unified toml rather than a ton of separate file.

.config is probably a better solution than a single top level toml, IMHO, but far more important is doing something unified rather than continuing the pollution of the top-level namespace, which merely obscures the project structure.

[1] https://snarky.ca/what-the-heck-is-pyproject-toml/

View on HN

That link is a really good read. Better than TFA, tbh. And also particularly well written

> An interesting side-effect of PEP 518 trying to introduce a standard file that all projects should (eventually) have is that non-build development tools realized they now had a file where they could put their own configuration. I say this is interesting because originally PEP 518 disallowed this, but people chose to ignore this part of the PEP xD We eventually updated the PEP to allow for this use-case since it became obvious people liked the idea of centralizing configuration data in a single file.

Hilarious and indeed interesting

This happens often in Node.js where you can put the configuration of some tools (e.g. babel, jest) inside package.json

I love pyproject.toml. I managed to get rid of setup.py, requirements.txt, requirements-dev.txt and MANIFEST.in, and I'm not far from having no setup.cfg soon.

All others stuff (CI/CD workflows, linters, formatters, code of conduct, funding, issue and PR automation, dependency bots, ...) moved to .github.

Now I have a clean and tidy Python project with a tiny 12-items root directory: https://github.com/kdeldycke/meta-package-manager

Python is unified on a requirements.txt file with pip, to manage dependencies.

I don’t think this is true. I’ve personally seen the following (and I’m positive there are others):

1. setup.py with self contained dependencies (this is my general preference)

2. setup.py loading from requirements.txt

3. Standalone requirements.txt

4. requirements.txt generated from requirements.in (or similar)

5. pyproject.toml + Poetry with a dependencies section

requirements.txt has a lot of deficiencies for package management (compare to the various features Yarn or Cargo offer). The biggest weakness is transitive dependency locking: even if you specify the exact version of the dependencies you want, Pip can still resolve to a different version of their dependencies, and even then there’s no verification that the content matches what is expected beyond the version number (other tools use a content checksum). This can cause a long tail of reproducibility and maintenance issues.

It’s also a tiny subset of what the parent describes.

It also can't separate dev vs test vs prod dependencies, for example. Maintaining separate files for those is not exactly a convenient solution

When I first set up my requirements.txt's, I usually set up a venv, install the things that I need and then run "pip freeze" to get a list of all deps including transitive deps, and put them all in the requirements.txts.

I do sometimes feel that people are making rube goldberg machines out of their package management in an attempt to avoid just writing down all their deps.

>The biggest weakness is transitive dependency locking: even if you specify the exact version of the dependencies you want, Pip can still resolve to a different version of their dependencies

Isn't that what pip freeze is for?

The trouble there is that if you freeze you can't unfreeze. A project needs the equivalent of both gemfile and gemfile.lock (unless you're using a system like maven that got things right the first time), but there's no standard way of having both of them in Python.

I strongly recommend https://github.com/jazzband/pip-tools to solve this. It provides a simple script to take a requirements file and "compile" a full specification of all transitive dependencies. You check both files into the repo, point pip at the generated file, and manually modify the other one. It means you often don't need to pin requirements manually at all, and the versions will be explicitly updated whenever you choose to recompile your requirements.

This is a big oversimplification. There are often multiple setup.txt files for different environments, virtualenv setup scripts, tox files, several different build config files, manifest files, and so on. Python probably has more magic config files than C#, which is quite an accomplishment.

Not "a" requirements.txt file, sadly. I've seen packages with different requirements for building, installing, and testing. All with their own damned txt file. Oh! And don't forget the duplication in setup.py -- you'll want to put your requirements there, too.

As far as I know, Pip doesn't have the notion of dev dependencies, so you need at least two requirements[.dev].txt files.

> As far as I know, Python doesn't have the notion of dev dependencies

Python doesn't have the notion of dependencies.

Pip doesn't have dev dependencies, poetry does.

D’oh! Thanks. :)

That's (the deps, not the dev deps) likely to change with pip 20.3! All hail the dependency resolver.

It's already available behind a feature flag.

https://pythoninsider.blogspot.com/2020/07/upgrade-pip-20-2-...

But there are usually several versions of python with several pips, possibly in combination with anaconda with multiple environments, a combination of named and rolling releases, and a SAT solver that's always slow but occasionally spontaneously decides to become extra slow and extend your build time by hours.

Oh and I forgot about setup.py. That too.

I tried to downvote this, but I think it’s already downvoted to the max. There is so much wrong with this pattern.

Please use something like Poetry to manage deps.

I never have a unified requirements file.

Reply on news.ycombinator.com