Hacker News Reader

The Arch Linux PyTorch 2.0 packages are great if you are looking for "cutting edge," as they are compiled against CUDA 12.1 now, instead of 11.8 like the official nightly releases. You can also get AVX2 patched Python and optimized C Python packages through CachyOS or ALHP.

But even Arch is still stuck on Python 3.10

Why people use conda? I couldn't figure out what it brings to the table.

I use pyenv on Linux and Mac and it basically works and with virtualenv plugin I can quickly switch environments to different Python versions.

I find it far better for DevOps. It is really easy to get the team up and running via conda environment files.

- Managing non-python / binary dependencies alongside python ones

- Managing a mixture of python / binary dependencies in local registries

- conda-forge managing builds of some really flaky binary python packages that are sometimes a nightmare to build locally

These are all legitimate reasons, but my personal experience (and perhaps preference?) is to use Docker for anything that is more complex than pip can handle.

> conda-forge managing builds of some really flaky binary python packages that are sometimes a nightmare to build locally

Yeah this is fair. Fortunately it's becoming rarer.

Well. To each their own, use cases differ so wildly it's hard to compare them.

The key audience for conda is ML/DS space, where most if not all packages come from either C/C++/Rust/Fortran and have to be compiled, while also requiring a consistent set of external C libraries like libblas, etc. As I said, some of those packages are a completely nightmare to build locally. Conda simplifies this by a lot in that you can just 'conda create -n myenv some=1.0 crazy=2.0 deps=2.0' and in a few seconds (if you use mamba and not conda) you have a working Python environment so off you go; no dockers, no local builds etc.

Honestly, I've found that conda has made operationalizing code very difficult. We've found it much easier to simply switch back to using pip, poetry, docker, and the standard OS package management tools rather than conda. Conda's dependency resolution is also quite slow and causes our builds & CI to timeout unless we drop in mamba.

Yes, precisely this. (I only do ML/DS work)

Can these hard-to-build-locally lib just upload a wheel to PYPI?

lol, the dependency hell of even using mamba can be large on old things, I have waited more than an hour for dependency resolution using both.

Seems docker is going from the frying pan to the fire. Have they added ‘resume download’ yet to docker? Over my slow DSL I can’t stand how docker makes me download 7G of images when I want to install something very simple, frequently it fails to do the download and I have to do it several times so it adds up to more like 28G of downloading and all that waiting.

I worked at one place where management was shocked when I told them the image build process would take 20 minutes on gigabit fiber up in Canada and we agreed to time it and I measured 18 minutes. Docker slows down “dev” to the speed of “ops.”

I don’t know how they did it but the data scientists could always find f-ed up Python images, you never got the same default character encoding twice, one time the default character set was Hungarian and I wonder how that happens…

Doesn't pip (with wheels) solve this all just fine? Or I'm missing something?

You're missing something. - Managing non-python / binary dependencies alongside python ones

pip with wheels doesn't deal with non-python packages. I used to be in a horrible locked down corpo laptop. Conda was invaluable in getting stuff to run, like chromedriver, etc.

Uh, I guess I do? I mean, there is https://pypi.org/project/chromedriver-binary/ (and a lot of similar packages)...

I honestly don't remember which one I've used for chromedriver when I needed it for my project, but I've surely installed all the stuff with "just" pip/poetry. Larger projects are typically packaged like this, with setup.py performing the downloads, while wheels solve the problem with Python libraries with native dependencies (e.g. how psycopg-binary works).

Maybe Conda makes it slightly more convenient, but I've always treated pip as the standard Python package management tool (it's a part of the standard library now, after all) and Conda was always "that weird non-standard thing some folks use for some odd reason" for me.

> Why people use conda? I couldn't figure out what it brings to the table.

Have you tried compile mkl, numpy from scratch?

Pip is a good alternative for numpy, it doesn't need conda. I think mkl is the same.

pip install numpy works

(though using poetry makes things less hectic in case of upgrades)

Afaik NumPy, SciPy, SymPy and Pillow are not managed/owned by Anaconda? At least here: https://numpy.org/about/ Anaconda isn't mentioned.

Ah, yeah they do have a Python 3.11 release, just not on anaconda. Okay, yeah, for a couple of years now there isn't a good reason anymore to use anaconda anyways.

Yes that's the issue! Most of the software is already ready, usable and just works... unless you use anaconda. Now that I think about it, is there some technical reason for that? I always thought it was mostly about stability, but I can't imagine python 3.11 being so unstable as to warrant waiting a whole year before even porting.

For one of my projects, conda ‘just works’ to get working with the GPU but following the instructions for pip doesn’t work. On the other hand there’s another package I am interested in using where I need to build out of GitHub and it’s a very different story.

I see myself as interested in commercial exploitation of transformers right now and I am delighted with the results. The first time I tried clustering all the Ukraine articles lumped together, all the sports were lumped, it runs 5x faster than my LDA-based clustering system and I think does a better job. With results like this I am happy to trade ‘cutting edge’ for convenience.

I have thought about a ‘path less followed’ in Python which is a truly sound package manager like maven for Python (as opposed to Poetry which I’m not sure is sound but it sure is slow) and I can say I like the way conda works I just would rather do it with wheels. One beef I have with conda is that the bzip2 files are slow to decompress and even over a DSL line I would trade a little more downloading for faster installs.

>>for a couple of years now there isn't a good reason anymore to use anaconda anyways.

I think of Anaconda envs as great back ends for Jupyter notebooks. There is still a place for it.

All the projects you listed are community projects that have grown over the years to become the backbone of scientific computing.

conda-forge, has evolved as one of the major conda community projects that helps release the latest releases from the projects you listed.

You can help improve the state of pytorch packaging on conda-forge too!

We've even released Pytorch 1.13 + Python 3.11 on linux and OSX! Give it a shot and let us know what you think!

edit: Link to the conda-forge pytorch development repository https://github.com/conda-forge/pytorch-cpu-feedstock

> It really sucks that anaconda always lags behind.

I usually just go for virtualenv (if python library versions are the only issue) or go for docker (if it's more than that). Both let you just use the latest and greatest without any friction. conda sits in a weird middle ground that I hate.

I use anaconda environments and install everything with pip in them. Am I doing it wrong?

That's basically where we are at for tons of our pipelines, but it kind of defeats the purpose since a dockerfile with a proper base image is basically equivalent at that point.

Conda is also really annoying to get working in a Dockerfile, because (if I remember correctly) of differences between login and non-login shells.

In general you shouldn't need to "activate" a Conda environment in the shell. Things generally "just work" if you use absolute paths. Something like this:

  COPY conda.yaml /env/conda.yaml
  RUN conda env create -f /env/conda.yaml -p /env/conda

  COPY foo.py /env/foo.py
  CMD ["/env/conda/bin/python", "/env/foo.py"]

What is a little funny is installing a consistent version of Conda inside a container, because the official Miniconda installers are rolling-release only. However you might be able to downgrade to your desired version of Conda after installation.

What's the point of this vs. creating your own venv (python -m venv .venv) and having a much smaller library of intentional tools?

Each environment can have its own version of python which downloads with everything else (venv would require separate system installs for this).

Can you clarify what you mean by separate system installs?

Today, I manage python via pyenv (local and global state) that let's me create infinite .venvs with different python versions.

Conda does the same job as Pyenv here, with the additional ability to install system-level libraries, compiler toolchains, etc.

i use venv this way. i download and compile specific python versions and install them in a non-system dir with all the other versions. then just run the specific binary to create a venv and it seems to work as expected.

Anaconda automatically handles things like making sure the correct version of cuDNN for your graphics card is installed. When I tried doing this myself with venv it was really painful.

There's nothing wrong with this. IMO, Conda is a general-purpose "system environment" and package manager that happens to be written in Python. The fact that its package ecosystem is oriented towards machine learning with Python is almost an historical coincidence.

Exactly my setup. I tried to use `conda install` few times, but every time after just few globally installed packages, conda SAT solver always struggles, and I now live with assumption that if incompatible package combination does not throw any error in dev environment, it is likely fine.

FYI the libmamba solver released last year is way faster than the classic one at modifying environments.

> *Though I guess Anaconda chewed more than it can handle w.r.t managing an entire Python universe, and keeping up to date. Conda-forge is already almost a requirement but using the official package (with pip, in this case) has its own benefits for very complex packages like pytorch.

Yeah, I absolutely adore conda, but they really need support.

Anaconda is really a solution for a personal dev box yes.

For containers a version controlled requirements.txt or Dockerfile is the way.

Realistically, Anaconda is a venerable solution from the truly god awful old days. The ecosystem is vastly improved.