Pip -t: A simple and transparent alternative to virtualenv in Python | Hacker News Reader

Pip -t: A simple and transparent alternative to virtualenv in Python

blog.zoomeranalytics.com

74 pointsfzumstein61 comments

I don't really get what this is supposed to solve.

- `virtualenv` works by creating an overlay of the `site-packages` in `$VIRTUALENV`, and change the shebang of scripts to use absolute path of python in the overlay. That way, not only do you get dependencies isolation, but you can also call directly your programs from outside the virtualenv and it will work.

What you propose is both bad practice (relative path in `$PYTHONPATH`) and rather annoying (you need to be in a specific directory to run everything). Trying to automate that will sure end up in pure hell!

- As in "you need to `cd` to _this_ directory then run _that_

- Or "WTF it worked on my machine when I was in this directory but now it doesn't"

Well as we mentioned, it's supposed to solve a situation where you don't want to use all the overhead of virtualenv.

The things you care about may be different from what others care about: I'm guessing don't care about Windows compatibility or you wouldn't talk about shebangs, and others may not find only running the scripts from the project root an issue (I don't for example).

Why is a relative path in the PYTHONPATH necessarily bad practice? It's not a rhetorical question, I am genuinely curious.

If you're happy with virtualenv then continue using it - all we have done is propose an alternative.

I'm curious, what sort of overhead are you seeing with virtualenv that leads you to use this/any alternative? I've used virtualenv + virtualenvwrapper on some seriously underpowered and low-on-space machines, and have really not noticed enough overhead to even make a blip.

I'm pretty certain virtualenv just changes some environment variables.

I found a writeup [1] on how to write your own virtualenv. It doesn't look like there's much overhead there.

I suppose one could justify not using virtualenv if they're on a system with limited resources or network access. I've known people who spent hours building a compiling python 2.7 from source on embedded systems because they rolled their own distro, and in that situation it would make sense to not want to mess with virtualenv if you just spent hours getting pip working.

1: https://www.recurse.com/blog/14-there-is-no-magic-virtualenv...

My impression is that the benefit of this approach is that it fits the mental model of NPM and Bower. Being able to manage packages across your stack using the same kinds of steps feels... organized. The relative path issue, however, needs some more consideration.

Yeah I think for me it's more about the intellectual overhead than performance.

With this you just have one tool, and it's just easier for me to understand what's going on compared with virtualenv.

Virtualenv is really not complicated. I think your time would be better spent learning how it works than just hacking some crappy workarounds to skip the problem.

It can be a pain when trying to distribute software to users (non-python programmers). What I usually end up having to do is distribute virtualenv with my code, and include a simple Makefile/build script to setup the venv. Invariably someone tries to then move that directory and everything ends up getting screwed up until I can tell them to run "make clean all".

I'm not too sure that this would end up being any easier, but maybe...

What I'm doing with streetsign[1] is including a download virtualenv step in the setup script[2]. The virtualenv gets set up in `.virtualenv/`, so most users won't even see it.

[1] http://streetsign.org.uk/ [2] https://bitbucket.org/dfairhead/streetsign-server/src/5b359a...

http://nuitka.net/

Does that work with Cython compiled dependencies?

You can build debian packages that way; dh-virtualenv (https://github.com/spotify/dh-virtualenv) makes it quite easy to pack a full virtualenv into a .deb.

A cross-platform solution that's not very different is just to invoke a virtualenv'd script with the version of python you want:

$ /path/to/.virtualenv/my_project/bin/python SomeScript.py

i remember reading virtualenv is like the simplest little program. what overhead are you talking about?

> Why is a relative path in the PYTHONPATH necessarily bad practice? It's not a rhetorical question, I am genuinely curious.

Among other things, what happens if your Python code changes its working directory, then subsequently tries to import a module? Consequences range from "doesn't work" to "introduces a security vulnerability by running arbitrary Python code from an untrusted directory".

Actually I don't believe that would happen. I think the relative PYTHONPATH gets expanded at startup and becomes an absolute path. Subsequent changes to the working directory wouldn't affect it.

That seems correct:

  /tmp $ PYTHONPATH='./test' /usr/bin/python -c 'import sys; print sys.path[:3]'                                                                                                                               
  ['', '/private/tmp/test', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip']

Good to know, thanks.

It's still odd for software to only work when launched from a specific directory, but not necessarily a critical security vulnerability then.

How is npm a model for anything? Just do a

  find . -type d -name node_modules | wc -l

from any reasonably large project's root directory -- the results will defy your sense of parsimony.

Packages which require other modules pegged to a specific version will install that package within their own node_modules directory, when they are already deep within a subdirectory of another node_modules directory. If two packages both require bar@X.Y.Z, they will both install that bar package in their own node_modules directory.

It is literally the most naive way to write a package manager.

What exactly is the overhead of a virtualenv? A few symlinks and a script which adds a few variables to your environment? Your calculation of what is and isn't complex is exactly backwards from the traditional Python perspective.

As a Pythonista, I can't help but object to the homebrew/npm school of roll-your-own package management. It eschews parsimony, deliberate architecture planning, and learning from the mistakes of other projects in favor of aesthetics and (sorry, I can't think of another name for this) hipster-DIYism.

Yes I have sometimes wondered about that. In fact another aspect of npm which I don't get is if you have multiple "instances" of a library (e.g. different versions, or even two copies of the same version) then any library global variables will not be shared across those instances.

But I don't know enough about JS development to say whether this really happens and/or if it is an issue (some pretty cool stuff has been done with JS so I guess it's possible to live with it).

BTW I share your dislike of hipsters, though I am a bit jealous those moustaches are just so cool.

> Why is a relative path in the PYTHONPATH necessarily bad practice? It's not a rhetorical question, I am genuinely curious.

Let's say you reimplement ls in Python. You then use your shiny new program to have a look at some downloaded files:

  $ ../utils/ls.py 
  ['hello.txt', 'os.py']

So far so good. Now let's try it with PYTHONPATH set to include the current directory:

  $ PYTHONPATH="." ../utils/ls.py 
  Boom! Your box is mine!
  ['no', 'files', 'for', 'you']

A caveat of the approach is that you must run the script from the project's folder.

This is apparently an unacceptable limitation for some people, judging by the responses to the article - however for others it's not.

I'll use the opportunity to moan about the state of packaging in the Python world. Why, after so many years, there is no way for me to ship software written in python, in deb format?

I end up doing some real crazy stuff:

* build in docker

* using virtualenv /usr/local/mypackage

* then manually copy over the init scripts "cp /usr/local/mypackage/bin/* /usr/local/bin"

* and debianize "/usr/local/mypackage/" and "/usr/local/bin".

Except it doesn't work:

* I need root do create the venv in /usr/local.

* This technique copies garbage like "activate" script to bin.

* virtualenv ships its own python binary. If the python version on host differ, it won't work.

* The workaround is to rewrite the bin scripts.

Total mess. All I need is a deb with some bin scripts, and all dependencies bundled. Is it _that_ hard...

I tried a few things myself, like fpm[1], rpmvenv[2] and lately pex[3] - but so far, I haven't found something, that I would call solid and simple. We have constrained deployment environments that do not allow most of the tools, that would ease the process on a development machine. Basically, I need a deb or rpm.

I have some hopes to wrap a pex into deb/rpm, but I would not call this approach simple.

That's unfortunate since Python is a wonderful language for many data-sciency tasks - Python makes things possible and pleasant, that would be a pain in other languages.

* [1] https://github.com/jordansissel/fpm

* [2] https://github.com/kevinconway/rpmvenv

* [3] https://pex.readthedocs.org/en/latest/

* Another tool: https://github.com/spotify/dh-virtualenv

We use dh-virtualenv and it's working out pretty well for us so far.

crohr's pkgr (pkgr.io) works well as well.

Did you try conda?

You can force virtualenv to include a specific python version

virtualenv -p python2.7 env_name or virtualenv --python=/some/path/to/python env_name

You've made the best argument for using pip like this, however. Instead of having to install a virtualenvironment and install dependencies and all of that, you could just ship your program with the pip -t libraries, though you'd have to include the path somehow on the install so python knew where to find those files.

Well, your user might have python in /some/path/to/python. My user my have in /usr/bin/python or maybe /usr/local/bin/python.

> Why, after so many years, there is no way for me to ship software written in python, in deb format?

How about using dh_python? And if you're building in docker anyway, why use virtualenv?

> All I need is a deb with some bin scripts, and all dependencies bundled.

Ah, that would be a problem. Python .debs are supposed to depend on separate python module packages, not bundle everything.

For what you're doing, have you considered turning your entire program into a single executable Python zipfile? See youtube-dl for an example.

I know that distutils supports to create RPM packages https://docs.python.org/2/distutils/builtdist.html#creating-...

I can't see why that cannot be adapted to .deb packages as well...

I don't know off hand if this handles dependencies, but I've used this in the past to create deb files:

    python setup.py --command-packages=stdeb.command bdist_deb

I'm pretty sure this works without requiring some special module.

What about dependencies? Setuptools assume all is in place. That's why you need pip/virtualenv. So with that you expect your host to have all the stuff.

Virtualenv installs some .so files in its thing, so venv directory is tightly coupled with the python binary it runs. Ie: you need to sort it out on the moment you build venv, and shipping venv directory is not correct.

> Looking at Javascript, tools like npm and Bower provide the easy, reliable and powerful package management capabilities which it feels like Python is missing. The key to their success? Both tools download a copy of the right versions of the right libraries, by default placing them in a special folder directly within your project's directory.

Isn't this exactly what virtualenv does anyway? Plus, with virtualenv you don't have to muck around in your $PYTHONPATH and you can still choose which Python version is installed for each virtualenv.

This article doesn't seem to explicitly mention any, but what are the compelling reasons to use pip over pyenv/virtualenv for project isolation? I guess using 1 tool to install dependencies and isolate your working environment is handy. I'm unsure if the overhead of virtualenv is so high that it offers a compelling reason to switch (at least for me).

I'd hardly consider virtualenv with virtualenvwrapper "overkill". It's a pretty elegant and incredibly easy solution to conflicting dependencies that is capable of being used on almost any other environment easily.

Elegance is always a relative term, but one could argue that having 0 dependencies is more elegant than having 2 dependencies.

Well 1 dependency, since this solution technically requires pip ;-)

oh, it's 2015 and people still don't know about pep-370, PYTHONUSERBASE and pip install --user :(

https://www.python.org/dev/peps/pep-0370/

That's for per-user isolation. This is about per-project isolation. A single user can have many projects with different dependencies.

that's what PYTHONUSERBASE (the environment variable) is there for!

Ah, I hadn't thought of using it that way. Seems like a working solution, even though it's not the problem that the PEP is meant to solve.

But you may want to keep 3 installation levels: global, per-user and per-project. If you use the second to achieve the third, then you no longer have the second.

PYTHONUSERBASE seems like a superior idea than any others to me. Wonder why it is not popular.

> forget about source env/bin/active and deactivate!

If the activate/deactivate bothers you, you can also just call the interpreter directly: env/bin/python ; you can also call any pip-installed binaries similarly. For example, env/bin/ipython if you have that installed. Or env/bin/pip.

I have a small utility[1] to magic that to "vpython" and "vipython" and "venv pip", respectively. (The last one is the general "execute this binary in the virtualenv" form; the utility makes some assumptions about how you name your envs, however, but also searches up the tree (I can cd into a dir, so I don't need to do ../../env/bin/python… or source the env.) I believe there's another implementation of this idea out there, but I can't find it at the moment.

(My vim is/was a bit python-heavy, and sourcing envs utterly breaks it if I attempt to start vim with an env sourced, since it gets the right vimrc, but the wrong pip installation)

[1]: https://github.com/thanatos/vpython

I started using pip -t ever since I started coding for AppEngine. I prefer it to a virtualenv becaus it works out a lot neater with the dev server. App engine's earlier approach was to use either virtualenv or pull all the dependencies' code down into the project directory but lately, the App Engine team have started recommending using pip -t instead.

Careful using relative paths in PYTHONPATH.

If you share a file system with untrusted users (like /tmp or an NFS mount) or extract an untrusted archive (zip, tar, rar) then you might inadvertently execute python code that another user placed there!

It's the same reason why you shouldn't use PATH=. in your shell.

This poor man's virtualenv doesn't seem like a good idea...

This is a very bad idea, for the same reason that relative directories are not on the regular PATH on Unix: On a multi-user system, or even a single-user system that sometimes has untrusted files on it, you can't safely use your tools if they can be overridden by whatever someone happens to have placed in the current directory.

For example, if you're looking through various home directories for some file, you might be doing something like:

  $ cd /home/foo
  $ ls
  $ cd /home/bar
  $ ls

If your PATH contained ".useful_stuff/tools" and the user bar happened to have a file called "/home/bar/.useful_stuff/tools/ls", that file has now been executed instead of /bin/ls, with your privileges. Your account has now been compromised.

Try this:

  [ptx@hn downloads]$ python3 -c 'print("hello")'
  hello
  [ptx@hn downloads]$ export PYTHONPATH="./.pip"
  [ptx@hn downloads]$ python3 -c 'print("hello")'
  You have been owned! Have a nice day.
  hello
  [ptx@hn downloads]$ cat .pip/site.py 
  print("You have been owned! Have a nice day.")

For this same reason, if you use Mercurial on a repository owned by someone else, you get something like:

  [root@host foo]# hg stat
  not trusting file /home/ptx/foo/.hg/hgrc from untrusted user ptx, group users

Microsoft had a related security problem a few years back where they would load DLL files from the current directory: https://isc.sans.edu/diary/DLL+hijacking+vulnerabilities/944...

This is a nice, simple solution to segregating python package environments. I personally prefer virtualenv, as I can navigate around the filesystem and open ipython from anywhere within the venv. I guess I don't see the usefulness of it so much because I have no problem using virtualenv.

The relative path is an interesting trick, I'll have to keep that in mind in the future.

Since the packaging wars end of 2012 to beginning of 2013 I was following relatively closely packaging and requirements tracking, and let me say this: We don't need another solution. We need the solution we have to work better. Pip+Virtualenv is enough to learn and handle and nowadays works well for most use cases. It's maybe not perfect but it gets the job done. Problems I still see: Most of all integration with the system's package manager; pip3 install -e will be overwritten by the corresponding pypi package if you increase the version, although the -e installed version is already the required one; sudo pip3 freeze doesn't work, at least on the current Ubuntu.

There are so many little details that need to be fleshed out in such an infrastructure tool, we can't solve all these boring problems multiple times.

There is probably an obvious answer to this, but a cursory google search turned up no answers.

The idea of having different multiple versions of dependencies on a single system is not exactly a new problem, and is solved by many other package managers. Is there some inherent weakness in python modules that forces them to be managed this way? Do the maintainers of pip insist on this for other reasons?

It seems like one of the bigger complaints I hear about python so I feel like it must be harder then it seems.

But this is exactly what virtualenv does...

1. You create a virtualenv 2. You enter the virtualenv 3. You install dependencies in it

Redo these steps for every isolated set of dependencies you want.

The solution proposed here does not solve anything. It only provides some half-working equivalent of virtualenv that works with a hidden directory to run your scripts...

The differences:

1. Your 'node_modules' is automatically created by npm. No need for a separate tool.

2. You don't have to 'enter' an npm environment other than changing to the directory where it exists.

These really aren't huge issues, but I can see it being an annoyance for some, especially people coming from Node.js to Python.

There is a more fundamental difference, which is what happens in a single project with a diamond-shape dependency with version mismatches, like this:

    A -> B, C
    B -> D == 0.1
    C -> D == 0.2

This dependency graph is handled easily by npm and other tools (at the expense of disk space, memory usage, etc.) and potentially small incompatibilities/gotchas, but is completely impossible to satisfy with pip.

IMHO complexity created by limited isolation facilities of virtualenv and the likes isn’t worth the resources you save by not just having a VM for each app you work on.

The complexity arises from differing system-wide dependencies that are out of scope of various package managers or virtual environments, and from development environment (assuming we’re addressing the need to have multiple apps co-exist on one machine for development) not matching production.

Not sure why it tries to replace virtualenv which is already obsolete in favor of pyvenv which is built-in.

Not everyone is off Python 2.x yet, for a wide variety of reasons.

Also have a look at vex: https://github.com/sashahart/vex

Unlike this pip solution, vex extends your PATH to include installed scripts. It also works well on Windows and is really easy to use.

Isn't it rather dangerous to have a "transparent" version of virtualenv?

This is somewhat similar to what I do ...

interesting! what do you do differently?

Packaging: if you have to do it, you should be building a bootable, distributable, OS instead.

Doesn't matter how many times others learn this lesson, I personally see it over and over.

Reply on news.ycombinator.com