I'm using 2080ti for my "at home" projects and going above CUDA 11.0 is indeed breaks lots of things - good luck trying to make something like colab/kaggle docker image for DS where you have TF+torch+sklearn+more combo.
Would you mind sharing resources you used to assemble your software/driver stack for 3090x2 setup?
https://docs.nvidia.com/cuda/cuda-installation-guide-linux/i...
This is nice because staying updated can be done through the usual apt commands, and you can have multiple CUDA versions installed by installing cuda-11-0, cuda-11-1, etc. Switching between them seems to be as easy as just making sure the appropriate /usr/local/cuda-<version>/bin dir is first in your PATH.
In general the NVIDIA documentation is pretty good, and I try to follow their Ubuntu-specific instructions whenever possible.