It's so weird to hear people who have problems with NVIDIA GPUs on Linux, because for me it's always been the opposite. I have had problems with AMD but never with NVIDIA.
But the user-space portions are probably more significant for performance than the kernel drivers. Here we have:
- r300 and r600 (open source OpenGL backend for older hardware, sits on top of the radeon kernel driver, not much development happening)
- radeonsi (open source OpenGL backend for newer hardware, sits on top of either the radeon or amdgpu kernel drivers depending on hardware version and kernel configuration)
- fglrx (closed source OpenGL driver on top of the fglrx kernel driver, both obsolete now)
- radv (open source Vulkan driver on top of amdgpu)
- amgpu-pro (closed source Vulkan driver on top of amdgpu) - not sure if there is also still a proprietary OpenGL driver but if there is no one should care since radeonsi works well enough
- amdvlk (open source dumps of amdgpu-pro without proprietary shader compiler on top of amdgpu)
Then you have different shader compilers which also significantly affect both shader compile time and runtime performance:
- internal compiler used by r600
- LLVM (used by radeonsi and amdvlk)
- ACO (used by radv and possibly radeonsi these days)
- AMD's proprietary compiler (used by fglrx and amdgpu-pro)
And for X.org you also have different display drivers (fglrx, radeon, modesetting).
Once the rewritten "amdgpu" driver came out, things got much better. The first few cards created after that (IIRC the Polaris GPUs, RX 400's), the situation reversed. I still have had occasional issues with various Nvidia cards (normally driver updates breaking things), but for almost a decade now, I have not had issues with AMD GPUs under Linux.
[0] Except for pro features while using workstation cards. You need to use a proprietary driver for those, but even those share a lot of code with the open source driver.
Even Sea Islands/Southern Islands were much better with amdgpu (but you have to use a module parameter to enable support).
Not completely true either, it eventually supported most of the normal 3d primitives but gaming performance was never a priority because there were few developers and they weren't employed by AMD/ATI -- which also meant that some cards would only reach full feature support after their EOL, sadly.
The amdgpu also driver benefits from a lot of the groundwork that has been done since. The radeon driver is older than kernel features like KMS (kernel modesetting) and GEM (graphics execution manager), and the LLVM-based shader compiler in mesa (userspace). I'd say that the radeon driver was actually the proving ground for many of these features, because it was the most capable open source 3d driver: The Intel 845/915 hardware barely supported 3d operations, and the only 3d-capable open source driver for Nvidia was the reverse-engineered nouveau driver.
Luckily, many people working on the amdgpu driver are actually on AMD's payroll these days.
I forgot that name "fglrx", probably a mental self-defense mechanism. Those were some bad times, trying to get different display outputs to work at the same time, guessing and testing values in xorg.conf, so on. There was some community utility someone wrote to try and help with installation, reinstallation, configuration and reconfiguration, but the name eludes me now.
I would edit my post to correct it, but it seems the edit window has passed.
Ah and hardware video decoding never ever worked again.
So much for the so called advantages of an open source driver.
So it is both driver changes and architectural changes.
There is also AMDGPU-PRO, which is the proprietary version based on AMDGPU. Used to be you'd need it for ROCm, but that hasn't been true for a while not. There really isn't any reason to use the "pro" version anymore, unless you have a some special proprietary app that requires it.
Open source GPU drivers are based on Mesa stack. So they share a common code base and support for things like Vulkan.
So it is sorta similar to how DirectX works. With old-school OpenGL drivers each stack was proprietary to the GPU manufacturer so there was lots of quirks and extensions that applied to only one or another GPU. That is one of the reasons DirectX displaced OpenGL in gaming... Microsoft 'owned' DirectX/Direct3d stack.
Well the open source equivalent to that is Mesa. Mesa provides APi support in software and it is then ported to each GPU with "dri drivers".
For gaming things have improved tremendously with "Proton", which is essentially Wine with vastly improved Direct3D support.
This is accomplished with "DXVK", which is a Direct3D to Vulkan translator.
This way Linux essentially gets close to "native windows speed" for most games that support proton in one way or another.
Which means that most games run on Linux now. Probably over 75% that are available on Steam, although "running" doesn't mean it is perfect.
One of the biggest problems faced with Linux gaming nowadays is anti-cheat features for competitive online games. Most of the software anti-piracy and anti-cheating features games use can technically work on Linux, but it is really up to the game manufacturer to make it work and support it. Linux gamers can sometimes make it work, but also they get flagged and booted and even accounts locked for being suspected of cheating.
The only issues you may run into if you distro doesn't include the firmware. e.g. This was the case with Debian 11 and you had to enable the non-free repo.
The only other problem you can conceivably have is card isn't supported by the kernel because it is too new. This can be fixed by upgrading the kernel. In Debian you can use a backports kernel, I am sure there are similar options in other distros.
When I was using my old 1080Ti, I had constant issues with the NVIDIA drivers. Acceleration didn't work on the second screen sometimes. There was some magic setting that would unset itself.
Mobile users suffer more problems then people with dedicated desktop GPUs, but it still gotten a lot better.
The one thing to be careful about AMD GPUs is that for most GPU OEMs AMD is just a after thought. So they get sub-par QA and heatsinks compared to their more popular Nvidia models.
It is best to go with card makers that only sell AMD GPUs, like Sapphire, PowerColor, and XFX. I am partial to Sapphire.