Hacker News Reader

Comment by robszumski | Hacker News Reader

As someone dabbling in ML for the first time, setting up the correct Python env has been a nightmare just to run some sample code. I am familiar with Docker and would love if this got more mainstream adoption.

Fighting with dependencies sucks when you aren’t intimately familiar with the ecosystem: python, node and other js, and even go.

View on HN

I haven't worked in ML, but I've used Python extensively for scientific computing. Setting up the environment used to be a headache, but Anaconda solved that issue. Are ML tools not available in Anaconda?

This is on my list of tools to investigate next. Glad I’m headed in the right direction.

I know you said "dabbling" but it seems like Anaconda is recommended by nearly every guide to ML with Python. What did you stumble across that recommended something else?

Move it to the top of your list. Anaconda is absolutely where you should start if you are trying to mess with ML/Predictive.

ML tools are available in Anaconda and is where we direct our data scientists until they are comfortable customizing further.

Have a look at pipenv - [https://docs.pipenv.org/], it provides deterministic builds and sane package management for python and is the officially recommended Python packaging tool now. If dependency management is your major problem, Docker might be an overkill.

P.S. -- Even if you are on Windows, whole SciPy ecosystem can be easily installed with pipenv, no need of conda for that now.

This looks pretty, but I've never personally had trouble with `pip freeze > requirements.txt`

What benefits does pipenv have beyond storing the hashes as well? In what way is it more sane? For example, does it get around the issues with, e.g., matplotlib being installed in a virtualenv?

I assume you are referring to the issues related to installation of external GUI frameworks in virtual environments mentioned here -- [https://matplotlib.org/faq/virtualenv_faq.html], I am afraid pipenv doesn't solve that any better than virtualenv’s --system-site-packages option -- [https://docs.pipenv.org/advanced/#working-with-platform-prov...].

If you are happy with your pip and venv workflow you can happily ignore pipenv, I personally prefer it because:

- It combines all the features of pip and venv with a better UI(in my opinion).

- I can specify my environments in more detail using Pipfile [https://docs.pipenv.org/advanced/#specifying-basically-anyth...]

- It is integrated with pyenv [https://docs.pipenv.org/advanced/#automatic-python-installat...]

- The promise of deterministic builds

Pipenv is only as good as the wheels that get installed. And the wheels are only as good as the assumption that your runtime/deployment environment matches the buildtime environment of the person who made those wheels.

Oftentimes with ML and compute-intensive workloads, performance is an aspect of correctness. It is nontrivial to get the right accelerated libraries, and the right GPU libraries linking against the right Numpy version for the particular notebook you're running.

If you are doing work just for yourself, or if you have total control over the deployment environment (and your collaborator's environments), then you might be able to get away with just using pip for these things.

Agreed :) when linking with native libraries conda definitely helps a lot. I was merely trying to point to the fact that after the recent release of SciPy we finally have binary wheels for Windows for the whole SciPy ecosystem.

What you need is the conda package manager (from anaconda python), in my opinion. Docker is very powerful, but is useful for other purposes (e.g deploying code) than simply grabbing a bunch of python modules and running analyses in a notebook.

I'm asking as someone who is mid level Python & DS experienced and out of real curiosity (not a "mine is better than yours" thought): Could you elaborate what in your opinion makes anaconda superior? I am frequently doing data munging & low level ML things and - so far - I am more happy with the pip side of things as a combination of virtualenv, autoenv [1], pip tools [2], pyup [3] and the rest of the eco system.

Standard procedure, takes two minutes and I have a Jupyter notebook up, smartly automated base processes for dependency management, full control of environment variables and can deploy/ integrate this in any other Python setup (if I were to port IPython code to pure Python).

Is this more a thing of each his own or am I missing a crucial advantage of Anaconda?

[1] https://github.com/kennethreitz/autoenv [2] https://github.com/jazzband/pip-tools [3] https://github.com/pyupio/pyup

Ease of use. I run outreach events and when you have sporadic internet access, being able to give new programmers an exe to install Python with a load of useful packages is a godsend. Also, it takes a lot of the pain out of setting up a Python environment on Windows when you need funny packages that require compilation (e.g OpenCV).

I've been to enough software carpentry talks where people nod and smile, and then promptly ignore the "how to be a responsible developer" advice. They don't care. They just want to run Python and plot some stuff or classify some data without worrying about multiple CLI tools. People seem to accept Anaconda because it comes across as a monolith and it's easier to understand for new programmers than virtualenvs and worrying about pinning pip installs.

That said, I don't use Anaconda on Mac or Linux.

I'm a fan of Anaconda, but conda sucks as a package manager compared to pip.

Could you elaborate on why you prefer pip as a package manager?

Lead conda dev here. Am definitely interested to hear more if you don't mind elaborating.

Does anyone see value in tool like michaelangelo from uber?

What if a data scientist could deploy models right from browser?

Check out Yhat. They were a Ycombinator alumni, and they offer that service. They were purchased by an enterprise software company called Alteryx, due entirely to the technical excellence of their product.

How? what is there besides running pip?

It can be a pain if you're running in a windows environment, even with pip.

Reply on news.ycombinator.com