Hacker News Reader

Comment by kristofferR | Hacker News Reader

So what is actually the solution?

Let's say I want to run Stable Diffusion WEBUI (Python 3.10) and Whisper (Python 3.11).

Last I tried to run both at the same time it was hell - when I downgraded Python for Stable Diffusion as per the instructions, Whisper stopped working.

View on HN

The short term solution is "relieving the packaging pain" link in the article.

The long term solution is described in the "What a solution could look like?" section of https://www.bitecode.dev/p/why-is-the-python-installation-pr...

The community is buzzing with attempts to fix those issues this year, so I’m hopping those posts will become obsolete one day.

Flask’s author is attempting something interesting with rye: https://github.com/mitsuhiko/rye

Trio’s author is drafting a spec for the equivalent of wheels, but for the whole python interpreter: https://github.com/njsmith/posy/blob/main/pybi/README.md

Not advocating to use them right now, but the fact is bootstrapping Python is finally acknowledged as one major cause of packaging issues and a priority to solve.

Those will likely cause more problems (as per https://xkcd.com/927/). While bootstrapping Python is a problem, it's a symptom, not a cause. The underlying cause is there is a substantial part of the ecosystem (of which the "data science" component overlaps with) that is coupled to hardware, the OS and external (non-Python) software libraries which doesn't align well with the single-language approach that a part of the Python community wants (vague gesture to the "web dev" side of the community). This drives the pip vs. conda arguments, the discussion around docker, nix, homebrew etc. You can't tool your way out of this divide over what the community wants (and while there are two vague meta-groups, there are really sub-groups which want different aspects of both).

It's worth looking at how some other popular languages (don't) handle this: from what I've seen, CGo is generally frowned upon, and so there is rewrites of libraries into go (Python does the same), or network/IPC connections are used to bridge the game; Rust also has the rewrite attitude, but for bridging to other languages there's build.rs (which calls other tools and wrappers), but those usually require you have the external library already installed (via some other means e.g. homebrew), or at least don't feature as a self-contained part of the cargo ecosystem (there's a cargo plugin which downloads a version of the library from somewhere); nodejs seems to either try to build from source (which fails if you don't have Python install for gyp), or download random binaries which sometimes work.

I use Nix shells & derivations to manage python versions and dependencies. The obvious disadvantage is needing to learn Nix, but the advantage is one system can track all the project's dependencies, not just the Python ones (e.g. you can also track the dependencies of Python modules written in other languages like Numpy).

You want something that manages different versions of Python itself. pyenv is a popular option for doing this, but I'm partial to rtx which is largely language agnostic and also handles Python virtual environments in a very elegant way.

https://github.com/jdxcode/rtx

Solution I'm using on windows is conda, think its called "miniconda" what you have to install. You'll need to make sure you activate the correct conda environment in the separate terminal windows to use both at once, works with CUDA.

But yeah everyones going to have their own way in that ecosystem.

The answer is probably some cross language/platform build system not written in python which is invoking python tools behind the scenes.

To me the solution is just use system python. If some packages do not work, then patch them if easy, or just avoid using if not.

Docker as last-resort solution.

you can use pyenv to have multiple parallel python versions installed and you can choose per virtual environment which version you want to use.

Pyenv doesn't seem to work on Windows, is the fork recommended? Or what about Pyenv on WSL2, will Cuda work well?

CUDA works great in a WSL2 vm. Highly recommended. I found that just sound doesn't work (any soundcard operation isn't implemented in the WSL)

Reply on news.ycombinator.com