Hacker News Reader

Comment by gtrubetskoy | Hacker News Reader

I'm surprised the article doesn't mention Anaconda, which is Python with all the things he lists pre-installed for you. I've been a fan for some time now: https://www.continuum.io/why-anaconda

View on HN

Another data point on Anaconda from microsoft's Azure ML team - we've been using Anaconda for the past few years and have been very happy with it. One key feature we like is that we can deploy a nearly identical suite of pkgs on Windows, Linux and MacOS w/o having to worry about mismatched versions, installation/update issues, etc.

Case in point - the Azure ML Jupyter notebooks run on a farm of Linux/docker machines:

https://blogs.technet.microsoft.com/machinelearning/2016/02/...

Once a model is built, it can be deployed for production/scale to the Azure ML backend which runs entirely on a farm of Windows machines.

Both environments have matching Anaconda distros running underneath which makes running on two different OS's practically a non-issue. In fact 95% of our users probably have no idea their notebooks & production site run two OS's (in large part thanks to Python, Linux, Docker, Anaconda, ... and lots of other great open source software).

I totally echo this. In my opinion, if Python was the Linux Kernel, Anaconda would be Ubuntu: i.e., it ensures that you are actually up and running within minutes and not struggling in the missing-header-files or dependencies-management hell.

Of course, there's time and place for eschewing Anaconda, especially if you are trying to ensure your installation and dependencies are minimal. That said, for most data scientists who are workaday *NIX users at best, Anaconda is such a productivity boost.

EDIT: Actually, my analogy needs further qualifications. You _can_ get into dependencies/missing-header-files hell in Ubuntu, but it usually doesn't happen in the first few hour/days. Both Ubuntu and Anaconda have great "first 5 minutes to 5 days" experience, and that goes a long way in ensuring the adoption of underlying technology.

I also think that Anaconda is great. However, I hope that in the future we could install numpy, matplotlib, jupyter, etc. just using pip.

I like Anaconda (and I recommend it as the easiest installation for data sci), but on OS X it is easy to install Python and relevant numerical packages with Homebrew and pip.

I did that earlier today:

pip install jupyter

pip install numpy

pip install scipy

pip install scikit-learn

pip install matplotlib

The only problem I had was with OpenCV, which requires manual make installation if you want the contrib package. The other problem was when trying to install scikit-learn, it requires manual pip installation of scipy.

The reason you can't is because of the C libraries levaaged must be installed prior to install numpy and scipy.

For example, you can't get through PyYaml unless python-dev is installed on Ubuntu. I am not sure if wheel would fix it but I don't think so.

Wheel is a binary distribution, so files would already be compiled and therefore python-dev would not be needed anymore.

Is wheel not default? Or play it elementary :-) how would I avoid this issue in the future?

I did not write anything worthy to send to PIP so don't know exactly but looks like it is up to the developers[1]

Anyway I just noticed that PyPI only supports binary packages for Windows and Mac OS X. Although, you could still generate wheels of packages that you use by using something like this:

    pip wheel -r requirements.txt

You can then install them with pip install <file> or (unfortunately I forgot the option, perhaps it was -i) you can use an option to point to a directory containing wheels and pip install to install the main package. It should use all dependencies in that directory as well.

[1] http://pythonwheels.com/

the issue with Wheels on Linux, and this is very much a Linux problem, is that the 'pre built' nature of libraries in wheels doesnt play nice with the raw chaos of Linux's package management + distro + kernel ecosystem. I raised the question of FreeBSD and Solaris based wheels at a PyCon when I was in a face to face discussion with someone more knowledgeable, and the answer was 'in theory that should work like Windows & OSX, no one has done the hard work yet.'

So yeah Linux is not the most friendly environment for Python Wheels.

I would always be worried with installing numpy/scipy via pip that I wouldn't be linking to BLAS/LAPACK correctly.

What do you mean? I have always installed via pip with zero errors on all three platforms.

The worry is not that it will throw an error, it's that it will silently link against lower-performance math libraries and make your code inexplicably slow.

Currently, pip install of numpy (and scipy) does not pick up MKL, which you would want if you do lots of linear algebra and FFTs.

conda supports pip installation within the context of an environment (much more gracefully, in fact, than does virtualenv). So you're not giving anything up by using conda in this regard.

I don't see the desire to have pip as the baseline. For me, the conda packaging is much more informative and placing everything you need for multiplatform support into an /info directory with a meta.yaml is a lot more effective than going through the steps of PyPI. conda also makes uploading and hosting on anaconda.org extremely easy.

Normally there is the whole "gee, I don't want to learn another package manager" -- but conda / anaconda.org is extremely worth it. It really is a major engineering step forward from the existing package deployment strategies in Python.

I even configure my travis.yml CI scripts to download Miniconda, create a conda environment from a requiremenets.txt, and then build and test my code via conda on the contiguous integration VM itself.

The only worry is how strongly tied conda and anaconda.org are to the future of Continuum. Given how much Continuum speaks of open-source work, one would hope that these projects essentially live independently (or that forks of them would) but you never know. I do admit that is a major downside.

I have anaconda installed at work/windows and home/linux. One minor annoyance is that it doesn't play well (at all?) with virtualenv. They worked so hard to get a huge suite of off the shelf third party libraries integrated, it's curious that they would come up with a replacement for virtualenv.

I'm halfway hoping that I don't know what I'm talking about.

http://conda.pydata.org/docs/using/envs.html

Glad to help :)

Right, I know about that. What I'm pointing out is that anaconda doesn't support http://docs.python-guide.org/en/latest/dev/virtualenvs/

It's nice that they have an env tool, but why didn't they just use the existing virtualenv?

Because using theirs means not having to recompile numpy (for example) with each environment.

Why doesn't anaconda's env work for you?

Not saying it doesn't work, just wondering why the duplication of functionality. Which you answered, thanks.

conda env also installs other dependencies for code. Really, the whole point of it all is that conda replaces pip and virtualenv.

Conda manages more than just Python packages, but other dependencies as well. For example, here's list of packages in a new environment I created with

  $ conda create -n hn python
  $ source activate hn
  $ conda list -c

  openssl-1.0.2g-0
  pip-8.1.1-py35_0
  python-3.5.1-0
  readline-6.2-2
  setuptools-20.3-py35_0
  sqlite-3.9.2-0
  tk-8.5.18-0
  wheel-0.29.0-py35_0
  xz-5.0.5-1
  zlib-1.2.8-0

setuptools and wheel are the only Python packages installed. The others are non-Python packages that conda is managing for my environment.

For all the talk of vendoring in package manager tools these days, I really, really like the way conda manages this stuff.

Reply on news.ycombinator.com