Python Virtual Environments – a Primer | Hacker News Reader

Python Virtual Environments – a Primer

realpython.com

115 pointsmjhea030 comments

I've switched to using conda for managing all of my local python projects and tools. Its an easier, more flexible interface and it can switch both the virtualenv and the python version. So no problem going back and forth from py3 to 2

Just install miniconda (without all the data science packages) and then use pip as normal

http://conda.pydata.org/docs/using/envs.html

Autoswitching when you enter a directory:

https://github.com/chdoig/conda-auto-env

How well do Django-based projects work with conda?

I'm running that in Vagrant anyway, but it would work just fine. Conda is just a well done wrapper for switching python versions and virtualenvironments.

You can also set ENVIRONMENT variables for each so that could be useful for specific django projects.

Most often you won't need to use pip, and `conda install ...` is preferable (for example, pretty much everything on PyPI is mirrored to anaconda.org, and if you ask them to support a well-known package that's not already available, it will be available extremely quickly, and chances are it's already available on some other anaconda.org channel anyway).

You can use pip from within conda as well if you want to, and you can refer to environment-specific pips directly, such as `~/anaconda/envs/my_env_name/bin/pip install ...` to install for the environment my_env_name.

conda can work with pip-styled requirements.txt files as well, and it's easy to either upload your own packages to anaconda.org, or create the conda build metadata (metadata.yaml) with instructions for how to get the package from PyPI if you prefer to only upload to PyPI.

There is an abundance of well-written tutorials and documentation for these things.

It'd be really great if conda knew how to fall back to pip, or if the conda skeleton -recursive / conda build / conda install cycle could be collapsed into one command

I'm not sure falling back to pip is a good default. You might rather be alerted to a package's absence on anaconda.org than to have it silently installed elsewhere, for example for use cases when you totally want zero pip installed packages.

But I agree there's a lot of opportunity for improvement around the configurability of this sort of thing.

It used to fall back to pip, but it was taken out. It's confusing if you typo the package name and then it tries to find it on PyPI and can't.

That's good to hear. Why is using conda install preferable ? Is it new infrastructure or something that you think might overtake PyPI ?

While conda is newer than virtualenv, pip, and easy_install, it's not really "new." Conda also is orthogonal to the source repository. As I mentioned, you can easily let conda manage the environments of your pip-installed packages.

One reason anaconda.org is better is that it's also Python agnostic. There is a main channel for R and R packages, so you can actually manage R environments the same way and R dependencies in a Python project. This could conceivably grow to include many things that pip alone can't help you with.

Having built packages both by using the necessary extra setting in a setup.py file, coupled with `python setup.py register` and the PyPI upload commands, and also via the meta.yaml and the anaconda.org upload commands, the latter seems more sensible to me.

Putting info into setup.py is a bad thing generally, especially if you do real work in setup.py, like with a complicated Cython package. Placing that info into meta.yaml offers better separation between data and code, and is remarkably easy to use in a cross-platform way with just `conda build`.

There's a lot more related to configurability and advanced options, but the above examples have been enough for me to fully use conda and to support other packaging as secondary when necessary.

What do you do about all the missing packages?

pip install <package>

Works perfectly well inside conda, and properly tracked. Packages don't have to come from anaconda/binstar.

Hmm, can someone mention the advantages of conda? Right now I use virtualenv with autoswitching, and I don't know what conda provides that's better. You mention switching the Python version, but virtualenv does that seamlessly as well, so I never had a problem.

yes, virtualenv switches to whatever python you used to create that virtualenv. but where did that python come from ?

conda installs pythons as well. like pyenv, but I lost a bunch of time trying to get that to work correctly.

everytime I used virtualenv I had to go look up things and many things still didn't work with the same ease as nvm rvm etc. There's virtualenv, virtualenvwrapper (to make it somewhat usable ? I forget); pyenv; pyenv-virtualenv (what?)

conda has always worked flawlessly for me without fussing.

Oh, it installs its own Python version? That's great, I needed 3.2 the other day but only had 3.4. I'll give conda a try, thanks!

I currently use GNU Guix to make virtual environments [0] (sometimes using containers) for software written in any language, which I find much nicer than a Python specific solution.

[0] http://www.gnu.org/software/guix/manual/html_node/Invoking-g...

I also think that Nix/Guix are the way forward... Guix really needs an installer/Linux package, though

Just yesterday I added a few tasks to the ansible playbook for my workstation to automate its installation:

https://github.com/berdario/dotfiles/commit/4e92cdb00dbe135f...

Small disclaimer: I'm also the author of Pew ( https://github.com/berdario/pew ), an alternative to virtualenvwrapper... and I'm working on a small program to try to automate further what pypi2nix/guix-import do to automatically package Python stuff for Nix/Guix... Until that'll be really easy, I think that whipping up a new pew virtualenv to try something out quickly is still the easiest thing (especially for people not used to Nix... I know: otherwise you can also use mutable virtualenvs on top of Nix)

We've moved to Docker Compose for local Python (Django) development.

Oftentimes when you need package management (pip) and multiple environments (virtualenv) you likely also have several services necessary for your projects (Python, Postgres, Nginx, Celery, et al.) and it can be easier for your team to run `docker-compose up` and have the whole stack running in lieu of setting up services manually and then initializing a virtual environment.

I much prefer taking the time to setup Docker Compose even for a simple project. I don't have to initialize a virtual environment, install packages, make sure the DB is running, sync with the DB, run the web server, etc. All of that work becomes automatic and I can focus on development instead sysadmin stuff.

Frankly I just use Nix, which is a little like having virtualenvs for your entire distribution and as such can handle non-python dependencies properly.

Virtualenv isn't supposed to handle non-Python dependencies.

Why use Nix instead of containers or VMs?

"Virtualenv isn't supposed to handle non-Python dependencies."

Making it kinda useless for anything like numpy or scipy.

"Why use Nix instead of containers or VMs?"

Well, why use virtualenvs instead of containers or VMs?

Because it's lighter weight and more straightforward. And doesn't require you to swap out your whole working environment every time you want a slightly different package setup.

Containers and VMs are a bad excuse for "I don't have a hope of getting the dependencies of my setup under control so I'm just going to ball them all up into this great big binary image."

Nix packages are incredibly flexible and composable in comparison.

I've been very happy with vex https://pypi.python.org/pypi/vex

I don't need the extra in virtualenvwrapper, just use the build-in venv module in Python 3:

$ python3 -m venv myvenv $ source myvenv/bin/activate

Note that pyvenv != pyenv, and the linked SO article appears relevant to the latter, not the former. Naming is hard.

Well, pyenv does have rather good virtualenv integration, and in fact I use that to make it easy to spin up a virtualenv with whatever Python version I happen to need at the moment.

I was puzzling over that exact thing for maybe 15 minutes. I've never used either one. Neither is on my path currently, even though I just installed python 3.5 with that option ticked. I can find pyvenv.py in my file system. Perhaps I'm expected to add it to my system path manually? Maybe I'll figure this out another day.

Using virtualenvwrapper and the plugin included with oh my zsh virtualenviroments are easy, but it is good to know what's happening.

I do not have a lot of experience, but I do have some, and in my experience miniconda is superior to virtual env.

Some packages don't install correctly with *conda.

you don't have to use conda to install. use pip as normal.

Just looked at it again. Seems like it's better than before. Will give it a try.

Reply on news.ycombinator.com