Conda is gonna work much, much, much better for these kinds of applications, as that's what it's mostly used for, i.e. scientific/numerical computing with C/C++ dependencies.
Use it in a container or a VM unless you enjoy re-installing your system from scratch.
Or better still, don't use it al all and let it wither away: these kind of braindead projects need to be put down with extreme prejudice.
But ... I seem to recall ... Conda tries to install GPU drivers does is not? ... Is that not the case anymore?
Because if it still does, your theory about "Why would your system care" and all that doesn't really hold water.