Specifying Installation Requirements for Python Projects | Hacker News Reader

Specifying Installation Requirements for Python Projects

python.org

55 pointsJetSpiegel18 comments

The deepest problem I see w/ the Python dependency system is the need to be able to declare substitute or alternative dependencies.

For instance you should be able to install a numpy that is compiled to take advantage of your specific CPU. You should be able to pick at install time which one you want WITHOUT patching the the source code directory in any way that would make it back to version control.

Do you have an example of how it would be solved in other languages?

In python I would create different requirements files together with freeze versions of those. One could contain the standard "numpy" and another would be "numpy @ my-custom-built-numpy-url". I haven't tried it and don't know how it would work in practice.

Yarn has the following: https://classic.yarnpkg.com/en/docs/selective-version-resolu...

This doesn't allow you to use an alternative index, but it does allow you to override packages that restrict their dependencies versions too strictly

Assuming you don't want to actually patch Numpy and just want to make sure it's compiled with support for processor features you have available like AVX-512 and SSE2 and what not, you can already do this by just telling pip to prefer source, which means it'll compile it at install time and the underlying build system will detect and compile in support for whatever processor features you can use.

I suspect it's the features of the BLAS that will matter, though, a lot more than Numpy itself. The Numpy build system by default prefers MKL, OpenBLAS, then ATLAS, and I don't remember what after that, and it's up to you to compile those with the features you need.

I DO NOT want to get anywhere near compiling dependencies at install time, that is how you make your Python build process entirely irreproducible!

I didn’t feel in control of Python builds until I had a wheels only policy. (For one things you can download about 2% of a wheel and get the dependencies to put into an SMT solver and globally solve the dependencies before you build anything. pip, poetry and similar systems use eggs and have to run setup.py to figure out deps which means they solver locally and eagerly and back themselves into corners that they may or may not be able to get out of.)

If I have to build from source I want to build a wheel, archive it, then reference the wheel in my repository. I don’t quite have the tooling to make it as easy as it really is.

I pretty much only do scientific computing with Python so take this with a grain of salt, but ever since I have started using conda I have never looked back. Especially for cross platform projects conda makes things much easier. I would love for the official Python packaging tools to be in a better state so I hope this helps.

Conda has the advantage that conda packages are typically repackaged from the Pypi package. Especially om conda-forge, they are curated by people with more experience in packaging, that are more likely to use appropriate version pinning for libraries. Furthermore, conda-forge has the ability to create a new build of the same package with improved version pinnings, which typically doesn't happen on Pypi for already released packages.

With the exception that the conda tool itself is so darned slow it can make you give up in frustration rather than wait another ten minutes to be told that the packages conflict.

So use https://github.com/mamba-org/mamba , which is an alternative client to conda and a million times faster. (If you are creating environments more than once per month or so).

I also do scientific computing all day every day with Python. I've used the built-in virtual environments using `python -m venv` for years on a project-by-project basis and have never had too much trouble. Yes, finding the right package compiled on Windows sometimes requires a trip to Christoph Gohlke's website, but beyond that it's never been a problem.

What does conda give you that is so much better than venvs?

I utilize a package that only distributes source or binary through Conda. Building it is very much non-trivial, so as to not go insane, I have to use Conda. Would really rather not use Conda, but bespoke packages + the trivial ability to define a specific Python version ($WORK servers use ancient RedHat with dated Python) has forced my hand.

Poetry feels better, if only because it separates out explicit vs transitive dependencies.

For personal use, most of the time, not too much. However I have encountered enough binary package conflicts when using pip that is just easier to use conda (or conda-forge).

On the distribution side, we are a small academic group that provides a niche, binary, scientific package that has quite a few binary dependencies (things like hdf, netcdf, etc.). We do not have the human effort to provide support for different distributions of Python, but it is very easy for us to say it will work if you use Anaconda with this environment on multiple OS's.

> `pyproject-lock.d`

Since this PEP is a draft, any comments/thoughts about naming the directory that stores lock files `pyproject-lock.d`? Is it normal to end a directory name with `.d`?

Reasonably so within linux filesystems where a file with that name might already exist, e.g. within Debian /etc/apt/sources.list is a file and /etc/apt/sources.list.d is a folder.

There's probably a more specific convention so I'll await murphy's law.

The dot-d notation is there for applications/packages to affect a subsystem without messing with that subsystems’ own configuration settings.

Debian makes good use of it but i found that it introduces a smaller problem of ordering (which of the app’s setting to apply firstly for that subsystem’s dot-d collections of apps’ settings.

But such conflict of inter-app configuration settings is seemingly rare and often best done directly into such subsystem’s own main config settings.

Any plans to add a toml reader to the standard library?

Exactly! I really don’t understand why they went with TOML — with pypackage.toml and now this — when there’s no stdlib TOML support. Given that it therefore requires an extra package, why not go with YAML? If they don’t care about optimising for human-readability, then stick to JSON.

As 3.10 is two months away from yeeting, this PEP665 would seem to be part of the 3.11 sizzle.

Packaging has long been a weaker suit for python. As with so many other language features (e.g. dataclasses) python seems to let a bunch of art settle in the wild and then offer a nicely refined PEP to capture and implement it.

Packaging PEPs aren't really tied to the CPython release cycle unless they impact CPython, like the deprecation of distutils. So if this PEP gets approved, you'll see projects using it much sooner than the release of 3.11

Reply on news.ycombinator.com