Show HN: Vex, a new way to run things in Python's virtualenvs
pypi.python.org
pypi.python.org
virtualenvwrapper is very powerful with all kinds of hooks but I just never used all that power and needed things like speed more. virtualenvwrapper provided a lazy loader but that would have various problems from time to time as you'll see on the issue tracker.
So then for a long time I just ran 'virtualenv' and source wherever/bin/activate and found that I really didn't miss virtualenvwrapper all that much, which was a surprise because everyone finds 'workon' invaluable, a position I do understand.
Later my wife taught herself virtualenv and then wrote like 5 lines of shell to do the same thing as virtualenvwrapper (the classic environment-modifying thing) which I found hilarious, and I wrote a version of that in zsh with a little more error checking to make me happy and just jammed it in my dotfiles. Then after some months of using that I decided this is nice enough to publish so I worked on rewriting it as portable shell functions with more error checking and was going to put it on PyPI.
Toward the end of writing that I was feeling pretty finished, when some last detail that I forget made me hacked off about modifying the current shell environment, so on a lark I wrote a prototype of the new idea of only modifying a spawned process's environment in about an hour (most of which was learning further about portable shell) and once it was working I looked at implementing more error checking for all the weird conditions that may arise in shell and I said, you know what, I have no reason not to do this in Python and totally cut the dependency on which shell is being used, who is going to use this without Python anyway?
Then I ended up figuring out how to do prompts and completion and pretty much had a publishable project, which was surprising.
It's still faster to directly run the virtualenv's Python or to use your own small shell function but while YMMV, for my uses virtualenvwrapper is a lot of code I will never use and I use vex every day.
Highlights of my todos: pydoc still doesn't work (the reasons for this are lame) and probably it will be nice to implement --make/--remove to substitute mkvirtualenv/rmvirtualenv (which would also let me dump a pydoc script in virtualenv's bin/ or Script\ dir to fix that problem...)
Are you using `pydoc` on the command line? If so, try pdoc: https://github.com/BurntSushi/pdoc. I wrote it as a tool to replace epydoc (strictly automatic documentation), but it works on the command line just like pydoc. (It can also generate HTML documentation with cross module linking that isn't crap. See an example here: http://pdoc.burntsushi.net/nfldb)
However, I still would like to eventually unbreak pydoc for users who might have bad habits like I do.
The issue is that virtualenv itself had an opportunity to fix this (with multiple patches submitted IIRC) and they decided to punt by implementing it as yet another shell function. So for me to unpunt without coupling to the shell, I pretty much have to dump a script into bin/. I guess maybe not that many people use pydoc anyway?
That would be my guess. IMO, pydoc isn't very discriminating with what it produces. The HTML is pretty ugly and the command line version just kind of vomits everything in the module to stdout. My idea with pdoc was to try and discover a module's public interface, and then display that.
At first I wanted to praise virtualfish [1] and its ability to automatically activate a virtualenv when I cd into it, but then I realized that vex's way does indeed make more sense. What do we do in a virtualenv? We basically just run pip to do various things, or run the program (in case it's an application and not a library) or test suite.
So yeah, I think I'm gonna switch. The added portability is a nice bonus.
edit: That one feature reminded me of Kenneth Reitz's autoenv (https://github.com/kennethreitz/autoenv).
I can't do anything to futz with cd, but it could be possible to reverse that and cd to a project directory when a virtualenv is activated, I am just not sure yet mechanically the right way to choose which one. Worth considering.
What I use that is analogous to your single virtualenv sourced at shell startup, is pip install --user. I tend to use that for command line tools like style checkers.
export PROJECT=`cat $VIRTUAL_ENV/.project`
export SRC=$PROJECT/src
[ -z $TMUX ] && cdproject && envopen
Where envopen is my own script that launches tmux (basically tmux -f $TMUXCONF -L $ENVNAME attach -t $ENVNAME with a few checks).This solution has a lot of flaws, though: it isn't robust and doesn't make sense for non-Python-based projects (my work is not limited to Django). In fact, I'm working on switching to dedicated VMs for my projects—looking for a nice solution currently. I suspect it may involve Vagrant, Docker, perhaps Chef.
Here's another project that attempts something similar: https://github.com/berdario/invewrapper
Here's a sortof-blog-post and some more discussion about the superiority of modifying a subshell rather than sourcing a bunch of janky shell-functions that modify your current shell: https://gist.github.com/datagrok/2199506
edit: after looking at the doc, my impression is that invewrapper seems to follow virtualenvwrapper's design more closely with a large number of subcommands with names similar to virtualenvwrapper's, most of virtualenvwrapper's options and features (except hooks and maybe some of the project stuff). I would describe the options as "comprehensive." I specifically wanted something with a dramatically simpler interface and feature set.
invewrapper also seems primarily or only intended to run a shell. I get personal use out of running arbitrary commands under vex as if it were sudo or something.
It is an improvement not to modify the current environment and couple tightly to specific shells either way, though.
One thing I do envy is the PowerShell prompt ;)
I just discovered your project, so I haven't tried it yet, but since the approach seems similar, it should work just fine on Windows (maybe with some tweaks)... and you can just add the Powershell prompt to your docs, no need to envy it :)
I agree that pew is intended as a virtualenvwrapper replacement, obviously you can get something similar to `vex env cmd` with `pew in env cmd`...
Unfortunately I've neglected a little bit the project lately: the most serious thing I want to do is rewrite the test suite, to be able to run it on windows.
Since you mention PowerShell maybe you use it quite often? I'd like to get some feedback on Windows (I added Windows support out of completeness, but I seldom use that OS), since I don't know anyone who uses pew on windows :)
BTW, I discovered some new tools to manage environments in the last year:
pyenv (I actually already knew rbenv), modules[1] and I even started to use Nix
These have not the same scope, and in fact I appreciate the latter 2 idea of using the same system to manage dependencies and versions for all different kind of tools. I don't have an opinion yet on what's the correct way to do these things so, in the meanwhile, even tools that are python-specific (like ours) have a niche to fill
[1] https://archive.fosdem.org/2014/schedule/event/hpc_devroom_e...
If you provide the full path to the python binary in the virtualenv, it executes with the virtualenv loaded:
/home/acjohnson/virtualenvs/foo/bin/python myscript.py
This is useful for crontabs, system service config, etc. which don't load a regular user's bash setup files.That eliminates the possibility that I'll accidentally run the wrong version of python - e.g. a system python or a python from another environment that didn't properly deactivate.
Most other issues are easy to deal with (e.g. non-existent virtualenv, being in the wrong folder), but the issue of accidentally using the wrong environment is awful because it can manifest itself with subtle bugs or weird exceptions rather than an obvious error.
So I can just go "pip install -u mercurial", or "pip install <some useful utility>" without worrying too much. And (my) "hg" is in my PATH -- and I can still run stuff in custom virtualenvs on a project basis, just as you say, by specifying the full path (./proj/venv/bin/python ...).
I've never understand why people use anything more complicated.
I do see occasions where activate is useful -- certain packages, I think scipy and pygame are among them -- are not only rather difficult to install via pip, but also tend to be very picky about their environment. Normally I get by by just installing those (and other packages with strong c/c++ library dependencies) via the package manager though.
Just like the author, I never used all virtualenvwrapper's features and was always happy to create and delete VEs on my own. The only thing missing from standard virtualenv workflow is easy way to activate the VE; so I wrote that on my own and been using it since ever. It's trivially small shell script but it does the work well, you can check it out at pypi:
This is also worth checking out if you just want 'workon' type functionality without the fuss.
function mkvirtualenv {
virtualenv "$HOME/.virtualenvs/$1"
if [ $# -eq 2 ]; then
vex "$1" pip install -r "$2"
fi
}
function lsvirtualenv {
ls "$HOME/.virtualenvs"
}
function rmvirtualenv {
rm -rf "$HOME/.virtualenvs/$1"
}Probably good to do some error checking on the arguments to mkvirtualenv and rmvirtualenv (sleepily hitting enter too early, etc.)
I currently use http://tudb.org/articles/2014/03/31/vpython/ for pretty much all of my python projects (also, i am the author of vpython).
Most of my projects are long running though (webservers, number crunchers, etc), so the extra miliseconds it takes to load is really not that important.
> Vpython is a tiny(-ish) bash script to help with your day to day virtualenv needs.
> * You don't have to worry about sourcing the activate script.
> * You don't have to point to your virtualenv path.
> just use vpython instead of python to invoke your scripts.
That's basically why I wrote vpython: I'm the author of this vpython[1], which isn't the same as yours, nor what the article is. (I suspect mine is a little more immature, as I've not worked on it for as long.) I guess similar needs beget similar utilities.
[1]: https://github.com/thanatos/vpythonWe take different aproaches though.
Repo link for reference: https://github.com/tbug/vpython
Mine expects you to know virtualenv and how it works, and also that you explicitly install it. Same goes for dependencies.
My needs where to avoid sourcing the virtualenv, so i wrote the tool to fix just that. Also, i built mine to resolve symlinks and be usable as the shebang in your python scripts, which is pretty neat when writing command line tools.