Hacker News Reader

Comment by afarrell | Hacker News Reader

You could alternately use conda, as much of the scientific computing community does: http://conda.pydata.org/docs/

I want to address several of the comments. I used to work for continuum, but I no longer do. I didn't work on the conda team, but as a part of the other teams at continuum, and at my new gig I'm a heavy conda user

1. linux package management is sufficient - this isn't the case. linux package management couples your library versions with your distro verison. Meaning if I want to do something like try out my code with the latest pandas, or run 2 jobs with different versions of numpy I'm out of luck

2. conda went ahead and forked everything without regard for the community - this is completely untrue. We sat down with guido for the very first pydata at google offices in 2012 to discuss many issues, packaging being one of them. Guido acknowledged that the python packaging ecosystem isn't sufficient for the scientific community, and we should go out and solve the problem ourselves - so we did. Honestly on this point, you should just read travis' words about the issue

http://technicaldiscovery.blogspot.sg/2013/12/why-i-promote-...

Scientific users need to be able to package non-python libs along with python libs (think zmq, or libhdf5, or R) we need a pkg mgmt solution that sits OUTSIDE of python for this reason. You can think of conda as a little in between place between virtualenv and docker

No, no no. No 'alternately'. The python community needs to pick ONE standard and, uh...standardize on it.

This isn't the issue.

Wheels and Conda are almost completely orthogonal. Conda is a higher level abstraction than pip. Wheels is operating at the lowest level of abstraction.

Multiple package management tools are relatively fine. Having multiple competing format standards (egg vs wheel vs lots of other options) would be a disaster.

(For reference, there's .deb versus .rpm, but also dpkg versus apt-get versus yum versus up2date versus...)

From a continuum.io blog post about Conda [1]:

> Having been involved in the python world for so long, we are all aware of pip, easy_install, and virtualenv, but these tools did not meet all of our specific requirements. The main problem is that they are focused around Python, neglecting non-Python library dependencies, such as HDF5, MKL, LLVM, etc., which do not have a setup.py in their source code and also do not install files into Python’s site-packages directory.

I haven't used Conda, but I totally get their main point here. Python packages sometimes depend upon non-Python packages. Any Python-specific packaging solution that cannot express that e.g. a Python package depends upon a native library does not really solve this problem.

I don't regard packaging, dependency management, and isolation as programming-language-dependent problems.

[1] http://www.continuum.io/blog/conda

I haven't evaluated Conda either, but the wheel vs egg distinction is categorically different from managing build chains for huge tools like Hadoop.

Replace "sometimes" with "nearly always" in the scientific world.

+1 that conda solves "most of our packaging problems" in the science/data science world (I've lived within it for 1.5 years, it really simplifies things). I use conda on Linux, Mac and Windows for r&d and deployment for tools including numpy, sklearn, matplotlib along with flask, pymongo etc.

Conda certainly sounds ambitious -- but I'm doubtful how useful it really is. As I see it, it mainly fills two needs: the lack of a package manager on OS X and the lack of a package manager on Windows.

Windows is getting a package manager now, and OS X arguably have the App Store.

Everywhere else (read: Linux/BSD) -- most of what's packaged by Conda, is already packaged by the vendor.

In short: I don't think it's the best way forward (but it may very well be the best way for many usecases/users right now and in the near future).

I agree.

There's probably also some element of "it isn't the best general solution, but it's super convenient for some particular group of people, e.g. python users".

A big part of Conda's usefulness is not system-wide package management a la apt or yum or whatever, but being able to have reproducible, isolated environments for scientific computing or other times when reproducibility is very important, and/or when you need multiple versions of packages for compatibility with other packages.

I believe it uses virtualenv for this, but I may be wrong. Regardless, having this feature (as well as no interference with the system Python) is one of the killer features for me, and others I'm sure.

This is a great point. Still, whole system snapshots in the form of a vm disk image, vagrant etc might be a better way to serve that use-case. Perhapssome form of appkiance levering something like guix or coreOS?

An additional benefit of Conda is that building or installing packages that require a C/Fortran compiler can be done without using sudo. This is really nice when you're working on a cluster or other shared computer resources, and don't want to have to bother the sysadmin.

As I understand it, it's still the same standard.

The distributions that I've used, unpack themselves to look just like a mainstream installation, including pip. So, if you have to install a missing package, you still do so in the same way. The distributions just give you a head start.

There are at least two major classes of people using Python.

On the one hand, there are people who write, distribute and deploy Unix-y-type software -- system utilities, web applications, service daemons, etc. These users write for users who are like themselves, or who are sysadmins of Unix-y systems. Their concern is with automatable, repeatable, command-line-friendly packaging, distribution and deployment. They benefit from wheels because it saves them a potentially-time-consuming build step during installation, and will generally prefer wheels or tarballs as a distribution format.

On the other hand, there are people who use Python as a scientific computing and number-crunching stack. Many of them do not write and distribute their own applications; they distribute the results of their code. Many of them run on desktop-type machines using Windows. What they primarily want is a single .exe they can download and run which will provide Python plus all the numeric/scientific libraries in one install, plus (increasingly) IPython Notebook as a way to share their work.

There is no single standard which works well for both of these groups.

> There is no single standard which works well for both of these groups.

Sure there is. A single exe that bootstraps python and leverage wheels to install what the scientific user needs.

In fact, with so many new tools comming to windows lately (as in the past decade) -- that exe could probably be a powershell-script. And an equivalent shell-script could probably do the same for more unix-y platforms.

Or one might make a script that builds an installer that bundles up things downloaded via the wheels framework.

I'm sorry if things aren't working well for you. The Python community already did pick a standard, which is PyPI, pip and wheels. All this is covered in PEPs. Unfortunately, the creators of Conda present their own tool to compete with the standard with a complete fork of all of Python's packaging, right down to virtualenv, which really only works for the packages that they wanted to support. That isn't the fault of the broader Python community. There is nothing anyone else can do to prevent that.

You can't use conda in many (most?) cases.

Specifically, there aren’t conda packages for every Python package. And if there are, they are behind their pip alternatives in many cases. Sometimes you then need to fall back to pip. However, it is impossible to install some packages into conda environments via pip. So basically the whole solution falls apart because the python libs you install via pip can't find C libraries installed via conda.

Edit: One example is: https://github.com/ContinuumIO/anaconda-issues/issues/177

Their solution was to include openssl into conda. Clearly, this solution doesn't generalize to other libs. Just things that use openssl.

The packaged distributions are definitely a blessing, when introducing beginners to Python. Even when package installation goes swimmingly, it still takes effort. And it's handy to keep just one big installer on a flash drive for putting Python on non-networked computers. Since I use Python for lab work, I often have it running on several computers at once.

Unless you want things compiled with SSL, then you're kind of hosed at the moment.

Which packages?

psycopg2

Why not both (at least on OSX and Windows)? Build packages where the users are. Even in the scientific space, many of your users build packages using pip.

Can you expand on why what makes Conda better?

Reply on news.ycombinator.com