Hacker News Reader

Comment by sealedservant | Hacker News Reader

Virtual environments are cool, and necessary, but at the same time, they are incredibly limited and I always get frustrated at the lack of features they should have. They are too fragile.

Nuking a whole venv when you mess up isn't really efficient.

They aren't portable. You need to package up your editable project for an offline system? Too bad, virtual environments use hardcoded paths and symlinks that will be broken when you try.

Want to convert the packages in your venv back to .tar.gz or wheels? There's no way to do that either.

View on HN

I've read "they're too/so/very fragile" so many times, but nobody has given me an example of how or why they consider them fragile, what steps do I need to do to break them.

Myself? I consider them ephemeral: I create my Makefile with a 'venv' target to delete/rebuild the virtual environment in case of changes. Some do take longer to rebuild, but with a global pip cache it takes much less time to rebuild once it's been done the first time.

Hardcoded paths and symlinks? Not a problem, I know that virtual environments don't travel; and since they're ephemeral if I need it at a different location I can always rebuild them.

Do you have an example for your third paragraph?

And I've done the last one, converted all packages to wheels, uploaded to an artifact server, and used those for production deployments.

I'm curious, really; not bashing you for having troubles with it, but I don't understand the aversion given my lack of blockers when using them.

> And I've done the last one, converted all packages to wheels

Did this require just downloading the packages again from the requirements.txt, or can pip do this? That could help me out quite a bit...

Use "pip wheel":

https://pip.pypa.io/en/stable/cli/pip_wheel/

The package should be installed first, then create wheels from all of them, finally upload them all to a repository for storage.

Can I ask you a question, what do you think is better approach: 1) publish packages as wheels or 2) publish apps as docker images (or docker-compose files or helm charts)?

Why some People prefer 1 to 2? I think 2 is more “production friendly” and universal across other languages and stacks (same approach used for java js ruby etc)

Vastly different use cases.

Publish a module as a wheel if you want to be able to use pip to install it in your servers, or anywhere really.

Once you've built out your virtual environment you can build your docker image and push it to a registry.

Your helm chart should include references to docker images that are getting deployed to a kubernetes cluster, among other entities that need to be built out for your app to work.

Those are all different layers of the deployment, independent of each other.

If someone prefers 1 over 2 (or viceversa), they don't understand what they think they know.

By publishing module you refer to situation when a single module is reused across different applications?

So if you dont have modules that used across apps, then you dont need to package and publish wheel?

or if your modules are already vendored in your main application module (monorepo) then no need to package app as a wheel?

The wheel, kept on an Sudafed repository, will let you lock down your supply chain, plus no need to rebuild every time it gets used.

I would build it once for whatever required architectures are needed, build wheels for them, amd use those for packaging up the final apps that need deploying (through docker, k8s, whatever).

Wheels can be production friendly if you know that you're only ever going to care about one specific OS family, and your app is pure python+stuff available on PyPi.

For internal use, where you can just pretend everything other than Debian based systems don't exist, it's ok, but I definitely see the value of docker, even though I like Snap a lot more.

I'm surprised nobody has ever made a "distro" where all the packages had a thin python wrapper and installed from a private Pip repository, so you could package your whole app and all dependencies with it.

How would you use the python `requests` library if it was packaged as a docker image?

I meant you package your whole app into container with all required libs, and dont publish your app as a package

Dockerfile and requirements.txt is everything you need to know to recreate environment, and this is how apps are usually ran in production. This is how you can create consistent recreatable and testable env and carry it from dev machine to test, staging, CI/CD, and prod environments

Never really had an issue with python apps not being portable.

Building regular python app? Just use official python docker image.

Using advanced CUDA stuff with nvidia cards? Just use nvidia’s docker image and forget about fiddling with configuring and compiling dependencies. It is all as easy as carrying dockerfile and requirements.txt

I dockerize, but using devcontainer.json, and this also plays well with VScode.

> They aren't portable. You need to package up your editable project for an offline system? Too bad, virtual environments use hardcoded paths and symlinks that will be broken when you try.

This isn't 100% true. If you carefully use the same python version and the same path for your venv and python then copying the venv between computers works perfectly. I've done it many times, actually last time I changed my computer I copied over all my projects and venvs and everything worked flawlessly.

Pip will cache the downloaded package tarballs to make repeated installs faster at least, but that's little help when you have a dozen venvs with multiple gigabytes of pytorch.

Time to containerize your pytorch app

> There's no concept of caching venv packages elsewhere

A venv's pip shares a systemwide cache for downloads.

Ah, that's news to me! Updated previous comment for accuracy.

I mostly use pipx these days for stuff I need on the PATH.

Reply on news.ycombinator.com