Hacker News Reader

> and PIP has been so much better than conda for packages.

> Is there something else I'm missing about conda?

Yes. You missed pip working perfectly well inside conda since day 1 (and integration is getting better). In the past, pip did not participate in conda dependency resolution (and I think it still doesn't, at least not perfectly). But it's not worse than using pip outside conda; and when you DO have a conda package, it is usually more dependable.

> I can easily run any version of Python, virtual environments are way easier to deal with, and PIP has been so much better than conda for packages.

Can you run the latest Python 3.8 on your old extended-support Ubuntu Server 14.4 that you can't upgrade because reasons, on pyenv+virtualenv? Genuinely asking; you can with conda.

> Can you run the latest Python 3.8 on your old extended-support Ubuntu Server 14.4 that you can't upgrade because reasons, on pyenv+virtualenv?

Definitely. If Python 3.8 isn't available from the package repository for ubuntu 14.04 you'll need to install it some other way -- either from source or from a binary package someone else compiled -- after which you can simply

    python3.8 -m venv venv

to create a python 3.8 venv. This also works the other way around. e.g. if you want to install an old python version on a recent system.

Of course anaconda provides python binaries itself which is mighty convenient but it's not like it's the only way to tackle the problem.

Ok, so that’s another good feature for conda - decoupling from the underlying distribution. It packages Python, gcc and more. To collaborate, you just have to distribute your environment.

I'm entirely mystified how people can get pip to work for anything that isn't either quite simple or is a shrink-wrapped product with carefully, and manually, selected versions of every component.

Anyway, conda isn't really aimed at either of these scenarios. It's aimed at people doing research, data science, etc, where you are likely to use a sprawling assortment of packages, many of bewildering complexity, and entirely without the time to tinker with which package version works with which.

The naive algorithm that pip use, which essentially is to grab the latest version of everything often doesn't cut it in these scenarios. Conda, with all it's warts, actually does dependency resolution, which it's why it's so much "slower", it does something completely different from pip.

As pip and conda can coexist today, I simply install what's not available in conda/conda-forge with pip, and have a much easier time manage all my dependencies.

If you do anything in data science, or similar, it's rarely a problem that things doesn't exist in conda, because they almost always do, which is not the case with all kinds of packages.

> I'm entirely mystified how people can get pip to work for anything that isn't either quite simple or is a shrink-wrapped product with carefully, and manually, selected versions of every component.

That might be down to which sorts of packages you're using. By far most projects I work on pip does entirely fine without manually specifying versions. I have rarely ran into version conflicts and when I have it's usually resolved by pinning a single package version.

The thing where I really see people trip with pip is when a package requires compiling a C module. Pip somewhat silently assumes you've got a working C compiler and all header files for the package. The error messages here are confusing and unhelpful for most python programmers. The whole push towards universal wheels is alleviating this problem but the current state of things is suboptimal.

Add to that the whole confusing state of creating python packages in general and you get a nice mix of dysfunctional packages and packages that'd work with the right headers. None of this is really pips fault and anaconda doesn't fix all of these issues either.

In my experience pip is mostly just harder to get started with but is afterwards a much better solution simply because pip doesn't hide anything from you which is a great boon when debugging packaging issues.

My perspective is it's useful for data scientists who are more likely to be 1) working on Windows 2) using packages with compiled extensions 3) less likely to have the system management background to properly deal with 1 & 2 in CPython.

I take Anaconda at their word that they are a different distribution of Python and conda is a tool to manage Anaconda environments, which includes more than just Python. I find it difficult to completely trust them as a for-profit company that is putting itself between Python programmers and the actual community package repos. I don't think there is any particular advantage to Conda unless you are support the use cases I described above, but good people disagree sincerely on it.

Agreed. It bothers me that Jupyter ecosystem trends toward it. pip + virtualenv + pyenv + poetry (not pipenv)

However, one of the main benefits I remember is that it is not just Python env/package mgmt. Data scientists can pull R packages too.

The issue I think is that it is just one thing as opposed to 4 different ones. Now I prefer pip and friends but I remember starting out it was hard to figure out and remember the interplay between all them. Especially with Windows in the mix conda was just much simpler.

Actually, conda + pip works pretty decently too. Conda lets you pull in different interpreter versions and manage them, install certain compiled extensions easily, and still install pip packages as normal when you need them. Recently, `conda config --set pip_interop_enabled True` was added as an option you could set to let conda be aware of pip packages when resolving dependencies.

I use Python for science data analysis. Anaconda has nearly all of what I need, batteries included. Even though I am able to do so, I don’t care to bother with managing my own virtualenvs and ensuring all the right stuff builds and installs on all the various machines I use.

If you are pure PURE python, and you have your system dependencies managed properly, then there might not be much point.

If you are primarily python but need to add C/C++/BLAS/LAPACK/MKL/Fortran, etc..., at least on linux and macOS, then you are going to want conda or something like it. conda with the conda-forge channel solves much of that.

There's other things out there which can solve it, notably Spack, possibly Nix. Or, you reduce the problem space by only building for a single linux and distributing containers, if that works for you.

Pyenv + Virtualenvwrapper is 100% the way, plus Poetry when it's time to get serious about packaging for a project.

If you go the route of Virtualenvwrapper (over just virtualenv), just make sure you install the Pyenv plugin version, as plain standalone virtualenvwrapper has annoying path issues that don't work well with Pyenv.

https://github.com/pyenv/pyenv/wiki/Plugins

For those considering alternatives to virtualenvwrapper, pyenv-virtualenv provides a very intuitive interface for managing virtualenvs similar to how python interpreters are managed via pyenv. I'd definitely recommend giving it a try.

conda makes cudatoolkit available. That can save a lot of time not having to figure out how to install CUDA and cudnn

Exactly this. Installing bhigh performance BLAS/LAPACK etc packages is very easy in conda.

Managing non-python software and its dependencies, for one thing. I’ve used it to package Qt apps, shell scripts, even JavaScript packages that were intertwined with non-js components (though that was painful).

For me it is that numpy by default will use Intel’s Math Kernel Library for linear algebra, on Linux, macOS, and Windows. It will also use FFTW.

That alone makes it very, very worth it.

Is there a reason MKL isn't shipped by default? I can't believe there is so much 'free' performance available that so many people are missing out on.

I used to try to use Conda but it is extremely crappy and does not give you anything over venv in case of using Python. On Mac we could not get Conda reliably working (tried with different shells but no dice).