Hacker News Reader

Comment by vimy | Hacker News Reader

I use anaconda environments and install everything with pip in them. Am I doing it wrong?

What's the point of this vs. creating your own venv (python -m venv .venv) and having a much smaller library of intentional tools?

Anaconda automatically handles things like making sure the correct version of cuDNN for your graphics card is installed. When I tried doing this myself with venv it was really painful.

Each environment can have its own version of python which downloads with everything else (venv would require separate system installs for this).

Can you clarify what you mean by separate system installs?

Today, I manage python via pyenv (local and global state) that let's me create infinite .venvs with different python versions.

Conda does the same job as Pyenv here, with the additional ability to install system-level libraries, compiler toolchains, etc.

i use venv this way. i download and compile specific python versions and install them in a non-system dir with all the other versions. then just run the specific binary to create a venv and it seems to work as expected.

Exactly my setup. I tried to use `conda install` few times, but every time after just few globally installed packages, conda SAT solver always struggles, and I now live with assumption that if incompatible package combination does not throw any error in dev environment, it is likely fine.

FYI the libmamba solver released last year is way faster than the classic one at modifying environments.

There's nothing wrong with this. IMO, Conda is a general-purpose "system environment" and package manager that happens to be written in Python. The fact that its package ecosystem is oriented towards machine learning with Python is almost an historical coincidence.

That's basically where we are at for tons of our pipelines, but it kind of defeats the purpose since a dockerfile with a proper base image is basically equivalent at that point.

Conda is also really annoying to get working in a Dockerfile, because (if I remember correctly) of differences between login and non-login shells.

In general you shouldn't need to "activate" a Conda environment in the shell. Things generally "just work" if you use absolute paths. Something like this:

  COPY conda.yaml /env/conda.yaml
  RUN conda env create -f /env/conda.yaml -p /env/conda

  COPY foo.py /env/foo.py
  CMD ["/env/conda/bin/python", "/env/foo.py"]

What is a little funny is installing a consistent version of Conda inside a container, because the official Miniconda installers are rolling-release only. However you might be able to downgrade to your desired version of Conda after installation.

Reply on news.ycombinator.com