Python Project-Local Virtualenv Management Redux
hynek.me
hynek.me
I bring this up because this is such a common topic to complain about, and there is this sentiment that Python packaging is completely broken and unusable. But for me, and I suspect a large quiet userbase, stuff just works fine, and I don't think this side gets brought up nearly enough. And for people not actively using Python, it's important to note that you don't need a rube-goldberg setup like in TFA.
Different OSes is a bit more annoying. I tend to just get VMs involved and do the build on the target OS.
About a month ago I set up a "homedir python" using pyenv, to hopefully solve this problem once and for all. So far it seems to be working great.
Though for projects I've been playing with poetry and rye the last year, basically just as syntactic sugar over venv, mostly just to try them out. They work fine, though I don't understand why "rye shell" is being deprecated.
I try to keep things simple.
pip’s depset builder is also non-deterministic up to its stated capabilities: at the stated capabilities dependency resolution is IIRC exactly NP-hard, it’s in that “you’re using a SAT solver but it will usually be fine” neck of the woods which is why more modern stuff (including I think apt and other Debian-adjacent stuff) use solvers now. You can limit the use of version ranges, you can pin like Bazel does with depsets (usually). This is why running pip repeatedly often solves the problem: you rolled a better character and/or it was hinted by a previous run.
Docker and friends can help but you get the usual Docker headaches, especially with qemu in the mix. I find OrbStack is just a strictly better option everywhere I’ve tried it, but docker over emulation/virtualization is rarely any kind of fun in the presence of accelerators.
So, I want to run an LLM or SD or some other now common thing, but I’ve got an NVIDIA box on x86_64-linux and a Mac on aarch64-darwin. What do I do? That sounds reasonable right?
Good luck. I went the whole way with Nix last time, and it solved lots of other problems (and in that world, problems stay solved until you are compelled to do a non-trivial update), but Python was still a nightmare.
This time I’m trying flox: and so far it’s been friggin awesome. It’s a little rough here and there for advanced users, but flox install pip, pip install torch/etc, and go? That works. On like, 7 os/cpu/accelerator combinations I’ve tried.
It probably sounds like I’m on their payroll but I’m legitimately just a very, very happy user who posts on HN in between excellent interactions with my bedrock env tool now. I hang out in their Slack but I don’t work there.
I'd say it is a good idea to pin python version in your project just like you'd pin any library - after all, there are changes to the standard library all the time. But how to do that in practice. Fuck me if I know a good way.
1. Just `pip` and `venv`; specifying dependencies via "requirements.txt" and "dev_requirements.txt", and tool configuration options in bespoke files. Too involved and with too many parts, yet it made do for my use cases.
2. `poetry` together with "pyproject.toml". Big quality of life improvement: commands like `poetry add`, `poetry build` make the process much less involved so one can just focus on actual development. Having all of the tooling configuration in a single place (that being "pyproject.toml") also helped with context switching.
3. `uv` for quick and dirty Python projects, `rye` (with the `uv` backend) for anything above that. In short, I think of `uv` and `rye` as faster siblings of `pip` and `poetry`, respectively. Once you get used to creating virtual and environments and installing dependencies in milliseconds or a few seconds it's difficult to go back to it always taking many seconds.
In hindsight, I would have been content just using `poetry`, but `uv` and `rye` do make for a faster and hence better developer experience.
P.S.: how messed up is it that only now we are agreeing on a convention for putting a project’s virtual environment into an in-project directory called `.venv`?
I much, much prefer virtualenvwrapper with it's style of keeping the environment outside the project.
This way, I don't end up with lots of huge directories hidden in the many many projects I check out.
The other thing that can't happen is accidentally adding the folder to version control.
I can have various versions of my project checkout out, and use the same venv between them.
I can quickly fire up a venv for the project with a different version of python.
I can quickly list all my venvs, and prune them as needed.
While people may say "we have infinite storage" my laptop only has 2tb and I fill it up.
But each to their own, a lot of people seem to like it.
I also use direnv, and the the virtualenvwrapper plugin for zsh, and have defaults - in this case it automatically defaults to the virtualenv with the same name as the directory I enter, but I didn't have to do any config for that to happen.
A few other ways I've seen embedded venvs interfering, off the top of my head:
* Polluting output from tools like `find`, `grep` and `tree` (yes, some have .gitignore options, but you have to remember to use them and some don't have the option at all, and often that's not appropriate anyway),
* Increasing docker build times due to not adding it to .dockerignore,
* Ending up in source packages somehow (e.g. serverless framework).
With inside project venvs what you see is what you get. VS Code can find them, no magic involved.
> my laptop only has 2tb and I fill it up
With venvs? Even assuming 4 GB pytorch/cuda/whatnot venvs that's a lot of venvs.
I didn't know about uv and PDM. I use pip-tools and there has been some work to adapt it to support multiple Python interpreters. My main gripe is it can't be installed once with pipx, for example, and just work. But it seems difficult. Perhaps uv suffers from the same problem. Interested to see how PDM deals with it.
> While direnv has built-in support for the standard library’s venv module, it’s glacial. virtualenv is much faster, but uv is faster, still.
I've never been bothered by the speed of venv. It does what it needs to do in milliseconds and it's not like I'm calling it that often. Why is the speed of it important?
Is there a standard method of making a "deploycontainer" from a devcontainer? Something that strips out everything not necessary to execute the application (however that is defined)?
I assume that by “package,” the author means library?
0. https://docs.python.org/3/glossary.html#term-module
1. https://docs.python.org/3/glossary.html#term-package
A non-canonical source claims broader meaning for "Library:"
A library is an umbrella term referring to a reusable chunk of code. Usually, a Python library contains a collection of related modules and packages. Actually, this term is often used interchangeably with “Python package” because packages can also contain modules and other packages (subpackages). However, it is often assumed that while a package is a collection of modules, a library is a collection of packages.
https://learnpython.com/blog/python-modules-packages-librari...
Package is a better term than library, because some of the things you can install with pip, such as command-line tools, don't really act like libraries.