I had that problem, too. It went away when I replaced my Nvidia card with an AMD one.
Nvidia's drivers suck but if you want to do AI stuff their cards are basically the only option until Intel and AMD complete ramping up their support for AI stuff. I wouldn't be surprised if two years from now AMD takes the crown with Intel being a close second (providing the best bang-for-the-buck) but for that to happen AMD needs to focus more on making actual implementations faster (e.g. PyTorch) and not just making minor incremental improvements to ROCm (and also make it so you don't need to use `amdgpu-install` to use it!).