Mapping the world of Python package management

Our solution to the infinite-backtracking pip dependency resolver was to instead rely on poetry to do dependency management via lockfiles. This way dependency resolution only needs to happen once, and oftentimes on CI through dependabot version bumps. This also has the advantage of ensuring that the exact dependency tree is mirrored on developer machines and on production servers.

I also helped work on python app dependency resolution in Homebrew. For apps (i.e. not libraries) written in python, Homebrew vendors the full dependency tree as formula resources. However this was a fully manual process for a while and relied on maintainers laboriously copy and pasting URLs from pyPI, checking requirements.txt and so on. We instead transitioned to a system that used pipgrip to do dependency resolution (and helped report a few bugs) so that all of this could happen automatically instead.

It's kind of sad that all of these are built outside of the pip team, but I imagine in a few years pip will be good enough to replace both of these use cases.

It's completely possible to download the metadata for a wheel without downloading the whole wheel.

Specifically a wheel is really a ZIP file, which has a superblock at the end that tells you how long the directory is, a directory just before the superblock. It is one or two range requests to get the location of the .dist-info/METADATA file and just one to read it.

Trouble is that this doesn't work for egg files, which are still around. In the example of a company that had Python systems that exceeded what pip could handle, we made our own private repository of wheels from eggs.

Good job modifying the title for HN, by adding the missing "Python" reference.

The article is still incomplete, of course, as it doesn't even mention that "packages" can be binaries, which introduces yet another dimension of complexity (architecture, dependencies, etc.) besides simple version numbers.

Python package management