Hacker News Reader

Comment by reubenmorais | Hacker News Reader

I don't get it... Can't you just use a virtualenv? Then there's no worry about namespace pollution or conflicts between dependencies of different projects.

View on HN

Installing a full virtualenv with all related libraries just to run one script is kind of ridiculous.

The server might have the wrong Python version available (2.x vs 3.x, multiple evolutions of 3.x with big features added in each point release). Et cetera, et cetera.

Or you could just do it with bash or a static Go binary and be done with it. Portable, works pretty much everywhere.

> The server might have the wrong Python version available

That's the point of using virtual environments, so that you can run the Python version and libraries that you need. Also, as of 3.3, Python ships with venv which means you don't need to separately install virtualenv anymore. It's all very portable.

You'd need to install the right Python version though, and even with something like pyenv you might need to install system packages.

It really isn't as portable as one might wish.

This is my exact situation right now. Our website is made in Django, and we use Pyenv to manage our python versions. We'll likely be going to route of a docker image/ pyenv install script. It takes trial/error to figure out some of the packages you need if they aren't installed already, a single binary would be far easier for sure.

Portable means something else. Portable would mean that I could ship the virtualenv. Python missed the forest for the trees.

I still need to have python installed on the machine and download all of the libraries needed.

Not all operating systems carry Python >3.3 by default, that would mean installing backports or unofficial repositories on production machines.

Or you can use a static binary that just works.

Why is it ridiculous? You'll need the related libraries either case, and by default virtualenv uses symlinks. The "full virtualenv" part makes it sound like it's heavy, or something like that.

If you do it in a language that provides static binaries or an equivalent, you don't need to pull dozens of libraries with possibly hundreds of files just to run a single script.

Imagine a situation where you'd need to perform an operation on a hundred servers, would you either transfer one static exacutable or build an virtualenv in every machine and download correct versions all of the related libraries?

If I controlled the environment, I'd probably build a Docker image and deploy that. Otherwise yes, This kind of extreme example isn't very representative of the initial situation though. And you're also talking about switching up languages, which isn't always possible or easy, especially in large teams or codebases.

virtualenv still requires internet access. You'd have to create a venv on destination machine and pip install your packages in. You can't scp a local venv onto destination machine and have it work.

I am an utter noob at Python packaging, but i would like to learn more, so excuse the basic question, but: why doesn't it work? And is there anything in the ecosystem which is like that, but does work?

I had heard that the Python packaging and deployment story had got a lot better in recent years, but this sounds like it still falls far short of table stakes.

I'm not sure why the Python community can't get it sorted out. We've been using Pipenv because it's one of a few tools that supports lockfiles/reproducible builds, and at the time we chose it, the Python packaging authority was advertising it as the official solution for Python packaging; however, after we were already invested in it, they backpedaled becaused the community realized it was very buggy and super slow (as in "it takes literally half an hour to add or remove or update any dependency").

We're finally getting around to migrating away, and we've settled on Twitter's github.com/pantsbuild/pants which is like Google's Bazel except that it's jankier in every way _except_ that Pants supports Python while Bazel doesn't (Bazel advertises Python support, but it doesn't work as the many Python 3 issues in their issue tracker can attest). The nicest thing about Pants is that it builds pex files (github.com/pantsbuild/pex) which are executable zip files that ship all of your dependencies except the Python interpreter and shared objects.

I'm still not very satisfied with this solution, and it's still far, far worse than Go's dependency management and packaging model, but it's a dramatic improvement over Pipenv, virtualenv, etc (gives us both reproducibility and performance).

Its not that Python packaging doesn't work, or that it is impossible to automate, but whatever you do it requires nontrivial time and effort to get right.

The point is that the static binary created by go is a single file that you copy into place and run. It is a lot simpler.

You just need a venv with the right architecture, which is fairly trivial. You can then copy the venv directly over.

See this repo: https://github.com/unixtreme/D3Edit. It has Linux/Mac/Windows virtualenvs and it works without any additional setup

I think what people are arguing is deploying a single binary is even more trivial than a "venv with the right architecture". Deploying one file is going to be easier than deploying a set of files.

I've done things in a way that it doesn't.

Basically build a virtualenv and then pack that into a .deb, install the .deb on the target machine and you have a self-contained package that requires no external resources to install.

That said, using golang is WAAAAY simpler. The virtualenv/deb solution, while it works, is very sketch.

So a virtualenv for every script I want to run? each of that downloading a full Python interpreter and whatever modules that Python script uses? How is this any better than a statically compiled binary? Other than useless additional work and bloat on the servers?

This is becoming a parody of software engineering.

People are now irrationally scared of using dynamic libraries, OS packages and even directories, like virtualenvs.

Instead, a simple solution is replaced with containers, or by rewriting tons of code in the new hyped language.

Are we trying to create job security through unnecessary complexity?

> Are we trying to create job security through unnecessary complexity?

As a python/go dev, I can assure you that go is not much more complex. And as a server admin, it's much easier to deploy.

It does have drawbacks when compared to python, but it can often be the optimal solution.

Also a Python dev. I do use VirtualEnv but the other thing is a huge performance boost for small performance intensive projects.

Isolation is merely one utility of containers.

This is the weirdest straw man I've seen in a while. Python, not Go, benefits most from containerization. Installing "Dynamic libraries, OS packages, and even directories like virtualenvs" on target machines is far more complicated than sending a single file. I say all this as the DevOps guy in a Python shop.

Reply on news.ycombinator.com