Comment by short_sells_poo | Hacker News Reader

Comment by short_sells_poo | Hacker News Reader

Parent Full thread

short_sells_poo

It's starting to look like the huge embarrassment that is python packaging and environment management. I work with the language every day, and I still have no idea what am I supposed to use between conda, pip, setuptools/setup.py, pyproject, meta.yml, poetry, pipenv, etc...

Conda is so slow that I sometimes wonder if we are being trolled by some cruel God of programming. Pip is faster, but version resolution is iffy.

Hell, even just assigning versions to python packages is nothing short of ridiculous. Do you use version.txt in the root folder and set it manually? Do you have it set from SCM? Which of the half a dozen packages do you use to have it set from SCM? setuptools_scm? Versioneer?

There are a set of tools in the python ecosystem that have basically no equal in any other language and these tools and the surrounding mindshare make python irreplaceable in the near term. The language itself is easy to learn and powerful enough to be able to do data analysis with ease. Good python code is easy on the eyes, which I personally consider an important aspect.

Outside of these tools, core parts of the ecosystem are basically an XKCD joke.

I tried switching to Julia as I find both the language and the ecosystem are vastly superior in their foundations. Unfortunately the maturity is not there yet, and neither is the mindshare. If I had to bet my career on adopting the language in a business setting, I'd not be prepared to do so. Which is a shame, because the situation turns into a Catch-22.

I feel like the problem is that pip is clearly the most popular, but philosophically the pip developers do not want to build a solution to all packaging and environment management into pip.

If I were pip dictator, I would try to make pip the one tool to handle all python packaging and environment management. In particular, that means pip would handle the management of different Python versions, different Python environments, native dependencies, running tests, making builds perfectly reproducible, releasing new versions of libraries, and creating new projects.

"Creating new projects" seems like it is not a big deal, but in practice I think that if there were simply a "pip new" command that set up a new project using the best practices advocated by the pip team, it would go a long way toward standardizing the ecosystem here.

You're kind of describing poetry. I think Python has a real future with solved packaging if Poetry can get the backing it needs at the language level.

Try mamba instead of conda [0]. It's really fast.

The first time you have to run it as:

    conda install -c conda-forge mamba

From then on, you replace conda with mamba. For example, if you are installing dask-cuda from the rapidsai channel you run it as:

    mamba install -c rapidsai dask-cuda

At this point, mamba is just so much better and faster, that it's the first package I install in an Anaconda environment.

[0] https://github.com/mamba-org/mamba

That's exactly the problem: instead of using conda you should instead use one of the myriad of others! It's balkanisation.

Unfortunately mamba doesn't support working behind proxies (so it's a non-starter in most corporate environments) and it's anecdotal but I've had it fail and crash much more so than conda does. It is faster indeed, but there's the other aspect as another poster points out: there's now Mamba and Conda, so instead of having one standard, we now have two. Plus, mamba can't be used for creating and managing environments, so you still have to do 'coda env create ...' for example.

Starting to? The Python ecosystem has been pretty behind other languages for more like 10 years now. I'm still forced to use it for work but it's becoming more and more likely that "no python" will be a hard requirement for my next job search...

What tools do you find irreplaceable?

There are actually clear answers to all your questions.

All the tools you listed do different things, except maybe poetry and pipenv, so you can pick whichever one you like, you're not supposed to do anything. You can have choice, illusion of free will, etc...

As to module version, there is a standard on how to define it in __version__: https://www.python.org/dev/peps/pep-0008/#module-level-dunde...

> There are actually clear answers to all your questions.

Strongly disagree. Conda, pipenv, pyenv, venv, poetry are all trying to solve the same problem (although conda tries to solve some other problems too).

Choice is not always good, this is why we have standards.

I would recommend pyenv. I understand why people are attracted to poetry, but pyenv arguably offers all of the same benefits that poetry has as well, and has better adoption.

> Strongly disagree. Conda, pipenv, pyenv, venv, poetry are all trying to solve the same problem (although conda tries to solve some other problems too).

Not really, some of these manage the issue of multiple system pythons (pyenv), while some manage isolated envs for particular projects (venv), and some try to be wholistic python project and dependency managers (pipenv, poetry, arguably venv + pip freeze, but that's not "wholistic"). Conda sort of tries to be all of the above as well as a bunch of other things (high performance options etc.)

Pyenv also has isolated envs for specific projects, poetry also tries to solve that problem.

There is a ton of overlap, denying that is disingenuous.

While there is overlap, there's also a ton of differences (pyproject for dependencies, dependency resolution, version locks).

I use both pyenv and poetry, and basically nothing else apart from the occasional call to pip.

I cannot imagine dropping poetry and only use pyenv, but I can imagine dropping pyenv (have been looking at asdf recently...)

Just use setup.py and __version__, ignore fashion until a winner emerges. Hasn’t failed me in ~twenty years.

This doesn't allow you to create conda packages, so you have to install things via pip. This can then cause headaches as you now have two mechanisms by which package versions are managed. Or you give up conda entirely and the scientific python ecosystem that comes with it.

Yes, most libraries have wheels, many others can be compiled, or installed from packages online.

Reply on news.ycombinator.com