Hacker News Reader

I do a lot of maintenance work and every time I've encountered a (complex) project set up in this way that's older than ~6 months, it's bitrotted from breaking changes in the dependencies.

The python ecosystem does not stand still and seems quite happy to introduce breaking changes in non-major versions.

To my mind, there are 3 ways to make sure your python project of today will work in 2 years time:

  1. Have no dependencies 
  2. Destroy your virtual environment and reinstall it every day for the next 2 years and fix anything that breaks
  3. Freeze your entire dependency tree where it stands and occasionally do large breaking change updates

All 3 options suck but #3 is the least worst

Is option 3 exactly what you’re supposed to do? Freezing your dependency graph and/or explicitly denoting what version of the dependency you want are your best bets for avoiding problems like this

It's certainly what I'd go for and what I recommend to everyone, but the GP disagrees:

> I never mention transitive dependencies in my requirements.txt file, just direct dependencies and rely on pip to install all transitive libs.

> You dont even have to freeze the version, just list the name and pull up latest version whenever you run pip upgrade

3 is exactly what you're supposed to do.

A lot of people will assume that specifying major version upper bounds on dependencies is what you're supposed to do, but I've seen this fail more often than freezing dependencies.

The problem with major version upper bounds is that if it's possible to write a test case for a bug, it's possible to depend on broken behavior. Changing behavior in a way that breaks users should be a major version bump, but that's not actually how people use semver and semver isn't really described that way either. It's described in a way that makes people think that changes in type signatures are the predominant impetus to bump major versions.

I mention this ceiling pinning footgun in the article. It's an enormous pain in the ass to explain to folks, and some software engineers I've met are totally incredulous that that's "not the right thing to do"

Poetry makes it 10x worse with its `^` operator

Definitely! And the article you link to in this section is great too

It is, but Python software tends towards large dependency graphs which will quickly accumulate CVEs, and so this strategy greatly upsets your security/audit people.

(Obviously patching vulnerabilities is good and proper, but automatically flagged CVEs are only ever _potential_ at best; in many contexts most of them are not actual vulnerabilities.)

Yes. But then any transitive dependency stays there forever even when not needed anymore.

Initial requirements (only first level, version ranges) and dependency resolution results (all transitive packages, exact versions or hashes) are two very different things and should be treated separately.

You can implement and maintain it by hand with two requirements.txt files but it's rarely done this way. And really at this level you're better off with a normal package manager.

I work in JS and the issues are the same.

It's why NPM creates lockfiles that can freeze the version of direct and indirect deps.

fwiw, we built an oss plugin for poetry on option #3 when publishing for repeatable installs to automate translation of published wheel metadata from lock file, also supports mono repos.

https://github.com/cloud-custodian/poetry-plugin-freeze

IMHO the problem with python package management isn't python package management. The biggest problem I've hit is when python depends on non-python - `pip install` failing because I need to `apt install libsomething-dev` is the big one, but also managing multiple versions of python itself (I count this as a dependency on non-python because the python3 binary isn't written in python - if it was, then pip+virtualenv could manage it like everything else). And also as you note there's just ecosystem churn.

If we could reboot all of computer science, I'd want one unified cross platform package manager with the same market takeover level as Git, and all languages designed to be aware of it.

No file system overlay stacking or anything, no file system hierarchy of different folders for different types of resources, just python style search paths and virtual environments, and no global environment allowed, ever.

But Python development is pretty nice at the moment so I can't complain.

What sucks is when you imply that your project will be unmaintained for 2 years, without any kind of update (not even basic security upgrades). This is a not a project, this is abandonware from day 1.

There was a time when you wrote a software for a version of the OS and it was done (except security fixes).

Obviously my profession gives me a ton of bias but in general it seems that for most people the squeaky wheel gets the grease and once a project is working it is ignored until it stops working again because there are only so many hours in the day and management has an infinite list of demands...

And then it breaks 2 years later and I'm called in because the person who wrote it left and no one else knows how it works because they didn't write any documentation etc etc etc

It's not like that simple 50 line Python script taking stuff from API A and stuffing them to API B is a full ass project with a JIRA project ID and a Product Manager.

It works, people forget it exists.

Then something breaks and I need to start fixing it. And the amount of work it takes to just update the packages is more than it'd take me to rewrite the whole thing in a language with a sane distribution mechanism (Go).

I go over the downsides in the article, but there is nothing fundamentally wrong with using pip and venv. It's just that if you've ever worked in other programming ecosystems, it would be immediately apparent that things could be significantly simpler and more reproducible.

Why does python need virtual environments? Why isn't it that you simply are in the correct environment when you're in your project folder? Why are there upwards of a dozen different config files for a project rather than just one standardized one? Why do you need to nuke your whole environment and recreate it just to remove one package?

These things are silly. They get in the way of reproducibility. They are security liabilities. There is a better way

> Why isn't it that you simply are in the correct environment when you're in your project folder?

I typically have several environments for my main project. Mostly I use a Python 3.12 version with the full set of optional third-party packages installed, but I also have a "clean" version with none of them installed so I can test error handling, as well as a Python 3.10 version as that is the oldest supported version for one of my third-party optional dependencies, plus a 3.8 version since I still support that version in my core system.

In that way I can quickly test code against the typical deployment cases before committing.

The project has about 500K lines of C code, mostly in one Python/C extension which takes over a minute to fully rebuild, but most changes are to Python code or other bits of Python/C or Cython code which are faster to rebuild, so I do editable builds for most of my work for the quick edit cycle.

I don't know how to nominate one of these environments as "correct".

(The full test suite against the various permutations of Python versions and 10 or so optional components is a mess. I've given up on testing all combinations as it wasn't proving worthwhile.)

Sounds like you may want to check out Hatch

How would that help? At the end I'm still using setuptools to build the extension, right?

So I'm stuck with the same core issues, but with a different front end?

To add to the complexity, I distribute my software as source, and I am wary of telling my customers to install another package in order to run the commands to install my package.

Virtual environments are cool, and necessary, but at the same time, they are incredibly limited and I always get frustrated at the lack of features they should have. They are too fragile.

Nuking a whole venv when you mess up isn't really efficient.

They aren't portable. You need to package up your editable project for an offline system? Too bad, virtual environments use hardcoded paths and symlinks that will be broken when you try.

Want to convert the packages in your venv back to .tar.gz or wheels? There's no way to do that either.

> There's no concept of caching venv packages elsewhere

A venv's pip shares a systemwide cache for downloads.

Ah, that's news to me! Updated previous comment for accuracy.

I mostly use pipx these days for stuff I need on the PATH.

Dockerfile and requirements.txt is everything you need to know to recreate environment, and this is how apps are usually ran in production. This is how you can create consistent recreatable and testable env and carry it from dev machine to test, staging, CI/CD, and prod environments

Never really had an issue with python apps not being portable.

Building regular python app? Just use official python docker image.

Using advanced CUDA stuff with nvidia cards? Just use nvidia’s docker image and forget about fiddling with configuring and compiling dependencies. It is all as easy as carrying dockerfile and requirements.txt

I dockerize, but using devcontainer.json, and this also plays well with VScode.

Pip will cache the downloaded package tarballs to make repeated installs faster at least, but that's little help when you have a dozen venvs with multiple gigabytes of pytorch.

Time to containerize your pytorch app

I've read "they're too/so/very fragile" so many times, but nobody has given me an example of how or why they consider them fragile, what steps do I need to do to break them.

Myself? I consider them ephemeral: I create my Makefile with a 'venv' target to delete/rebuild the virtual environment in case of changes. Some do take longer to rebuild, but with a global pip cache it takes much less time to rebuild once it's been done the first time.

Hardcoded paths and symlinks? Not a problem, I know that virtual environments don't travel; and since they're ephemeral if I need it at a different location I can always rebuild them.

Do you have an example for your third paragraph?

And I've done the last one, converted all packages to wheels, uploaded to an artifact server, and used those for production deployments.

I'm curious, really; not bashing you for having troubles with it, but I don't understand the aversion given my lack of blockers when using them.

Can I ask you a question, what do you think is better approach: 1) publish packages as wheels or 2) publish apps as docker images (or docker-compose files or helm charts)?

Why some People prefer 1 to 2? I think 2 is more “production friendly” and universal across other languages and stacks (same approach used for java js ruby etc)

How would you use the python `requests` library if it was packaged as a docker image?

I meant you package your whole app into container with all required libs, and dont publish your app as a package

Vastly different use cases.

Publish a module as a wheel if you want to be able to use pip to install it in your servers, or anywhere really.

Once you've built out your virtual environment you can build your docker image and push it to a registry.

Your helm chart should include references to docker images that are getting deployed to a kubernetes cluster, among other entities that need to be built out for your app to work.

Those are all different layers of the deployment, independent of each other.

If someone prefers 1 over 2 (or viceversa), they don't understand what they think they know.

By publishing module you refer to situation when a single module is reused across different applications?

So if you dont have modules that used across apps, then you dont need to package and publish wheel?

or if your modules are already vendored in your main application module (monorepo) then no need to package app as a wheel?

The wheel, kept on an Sudafed repository, will let you lock down your supply chain, plus no need to rebuild every time it gets used.

I would build it once for whatever required architectures are needed, build wheels for them, amd use those for packaging up the final apps that need deploying (through docker, k8s, whatever).

Wheels can be production friendly if you know that you're only ever going to care about one specific OS family, and your app is pure python+stuff available on PyPi.

For internal use, where you can just pretend everything other than Debian based systems don't exist, it's ok, but I definitely see the value of docker, even though I like Snap a lot more.

I'm surprised nobody has ever made a "distro" where all the packages had a thin python wrapper and installed from a private Pip repository, so you could package your whole app and all dependencies with it.

> And I've done the last one, converted all packages to wheels

Did this require just downloading the packages again from the requirements.txt, or can pip do this? That could help me out quite a bit...

Use "pip wheel":

https://pip.pypa.io/en/stable/cli/pip_wheel/

The package should be installed first, then create wheels from all of them, finally upload them all to a repository for storage.

> They aren't portable. You need to package up your editable project for an offline system? Too bad, virtual environments use hardcoded paths and symlinks that will be broken when you try.

This isn't 100% true. If you carefully use the same python version and the same path for your venv and python then copying the venv between computers works perfectly. I've done it many times, actually last time I changed my computer I copied over all my projects and venvs and everything worked flawlessly.

> Just nuke and recreate venv from scratch using only what you need.

That's great until you have dynamic dependencies which means you can run apparently happy yet fail later on an import. This means you have to take special care generating your dep list.

Thats what pytest is for, it is important to have tests that cover your most important code paths so these these dynamic imports are ran.

Due to dynamic natire of language, coding in python without strong test culture is masochism and full of footguns anyways

> I never mention transitive dependencies in my requirements.txt file, just direct dependencies and rely on pip to install all transitive libs.

> You dont even have to freeze the version, just list the name and pull up latest version whenever you run pip upgrade

This gets you into situations when you need to release an urgent one-line hotfix into production but your docker builds are broken because some version changed. Stable reproducible build are an absolute must if you ask me.

Pip is mediocre for vendoring non-Python or binary dependencies. Wheels are OK, but sometimes there are still problems.

Conda solves this pain point by being a general purpose package manager.

Poetry is for the more reproducible builds people. It's a little better, but it's not great enough to get people to switch.

> Can people explain why venv&pip is a bad solution that doesnt work for them they have to resort to other package managers?

It's very simple:

1. As you said yourself, you don't need venv at all. 2. The format used by `pip -r` (colloquially known as "requirements file") is not a defined standard. 3. pip is not a package manager, but a package version resolver and installer.

Pulling the latest version without freezing is how you Break Stuff. Every python developer seems to be addicted to breaking changes.

Nuking and recreating, freezing, etc are all easy minor tasks but nonetheless add a few lines each of code.

Auditing for security issues would probably need some other package or other.

And with virtualenv, you can have an activated environment. It's stateful and your scripts have to then take into account state. Poetry run doesn't have that issue.

That's enough minor annoyances to make me not want to ever use anything but poetry plus the audit plugin and freeze-wheel if needed.

> This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from scratch using only what you need.

There was (or still is, I wouldn't be surprised!) a time when the AWS CLI and the AWS Elastic Beanstalk CLI had incompatible versions of some dependency and couldn't be installed in the same venv. Not fun.

I also use venv and pip and I'm happy with the solution. I don't get the knock on JavaScript node modules though. Why wouldn't you want to list out your dependencies and their versions? It can be greatly helpful and I don't see how it contributes to bloat

I use the same workflow, and works for me very well.

It would be cool, though, to have a wrapper around pip that adds any packages I install to the requirements file. That way I don't have to do it manually.

I actually would love something like `pip generate-reqs` where pip goes through every .py file in my app and looks at whats being imported and writes these libraries to requirements.txt, together with all transitive dependencies from the currently installed `pip list`

pip freeze Is what you're looking for

pip freeze lists all possible installed packages with exact version.

I want only the manually installed packages in exactly the way I installed them.

So pip install numpy>=3.2 should put numpy>=3.2 in requirements.txt. and pip install numpy should put numpy in requirements.txt.

pip is so easy, but unfortunately i've found that if you add package signatures to requirements.txt, pip chokes on it when installing it later. And subdependencies aren't always named perfectly, e.g. they might specify ~=1.4, and a subdependency that what was once 1.4.0 is now 1.4.27, and incompatible or compromised.

conda is so heavyweight installing whole pre-approved builds. and the command line options I find extremely frustrating.

I need supply chain security and perfectly reproducible builds, so poetry was the only real option.

wouldn't `pip freeze > requirements.txt` solve your problem? It will list everything currently installed, including transitive dependencies at currently installed and working versions