Hacker News Reader

Comment by dheera | Hacker News Reader

The real issue is having to keep multiple copies of CUDA on a system to satisfy all the different moving parts that each want different CUDA versions and CUDNN versions, and those different CUDA installers fight over and uninstall each others' nvidia driver versions because CUDA is shipped with the nvidia driver, bundled and listed as an apt dependency.

My system changes from nvidia-525 to nvidia-535 to nvidia-520 to nvidia-515 on a daily basis because I need to reinstall a different CUDA version just to try some new paper's code.

PyTorch did it right, it now ships with its own CUDA and doesn't take a shit about version what you have in /usr/local. Everything else should do the same.

View on HN

The solution is to install the latest CUDA driver (which is always 100% backward compatible) but have multiple version of the libraries in /usr/local. Granted it can get messy with various package manager doing their own things.

Usually the CUDA versions end up uninstalling other CUDA versions and drivers due to package manager issues.

I wish nvidia would just release a `sudo apt install cuda-all` that just keeps you updated with ALL possible cuda versions simultaneously. I know it would be 20GB but that's fine, it's a drop in the bucket compared to the datasets I play with.

If you use the run files rather than the package manager files it's always installed in completely separate folders inside /use/local/cuda.

What you're describing with an "all" install can somewhat be accomplished with containers right now and none of the dependency problems.

But containers, X forwarding, visualization, and OpenGL is a complete shitshow. I have apps that work outside of a container but freeze the UI that the mouse pointer won't even work when run from within a container.

+1 for runfiles but beware of the multi-gigabyte installer + installed footprint. Not complaining per-se, just plan for it.

Problem with runfiles is I've run into other .deb packages that I'd like to use that are marked as dependent on 'cuda'.

Using a runfile is ditching the package manager on a package-managed system, instead of using the package manager correctly.

> now ships with its own CUDA and doesn't take a shit about version

Bundling system dependency in python software is always a giant shit show. I would not name that "Everything should do the same"

Any other python library (randomly cuPy) that also ship its own Cuda, with its own different version, will segfault happily because you have duplicated symbols in a single process.

So randomize the symbols with appeneded UUIDs

What you just quote is more or less what the GNU glibc is doing with symbol versioning which is an other example of giant shit show and an ABI nightmare.

And:

(1) It is a solution only Nvidia can provide: Cuda is a binary. And they will certainly not do that just to please the python community.

(2) That just create an other set of problems just because python packaging sucks in the first place.

Lets just solve the initial problem, shall we ?

> PyTorch did it right, it now ships with its own CUDA and doesn't take a shit about version what you have in /usr/local

Is that even legal ?

Up to my understanding Cuda is covered by an EULA license that explicitly require end user agreement.

"Meh" who cares. It works.

Don’t install the drivers with CUDA? Don’t rely on distro-packaged CUDA?

NVIDIA's official CUDA is packaged with their damn driver.

Hi, you can choose not to install the driver with a CUDA install, and to download the driver separately.

Only if you use the runfile, but that breaks yet other packages that specify 'cuda' as an apt dependency.

Your 'cuda' packages should use '>=' not '=='

e.g. 'cuda-12' should depend on nvidia>=525 NOT nvidia==525

That's a fair point. I'll look into it.

Hi, I looked into this and it seems that it is indeed using >=

Depends: cuda-libraries-11-8 (>= 11.8.0), cuda-drivers (>= 520.61.05)

Hm no, what you actually have is cuda-libraries-11-8 depends on cuda-drivers and cuda-drivers depends on cuda-drivers-520 = 520.61.05-1 where you have the version number repeated in the name. That is a really bad pattern for things intended to be backward compatible, as multiple installs of cuda will overwrite "cuda-drivers" and each is pegged to a certain "cuda-drivers-X" where X is part of the name. This results in some really weird dependency hells if you try to install more than 1 CUDA version at the same time depending on the order you install them.

    $ sudo apt show cuda-drivers
    Package: cuda-drivers
    Version: 520.61.05-1
    Priority: optional
    Section: multiverse/devel
    Maintainer: cudatools <cudatools@nvidia.com>
    Installed-Size: 7,168 B
    Depends: cuda-drivers-520 (= 520.61.05-1)

Reply on news.ycombinator.com