With a practical example:
1 - You boot up your Linux base system.
2 - You fire up KVM with GPU PCI passthrough with a Windows guest to play some games.
3 - You shutdown windows.
Now you can't use the GPU anymore, the module was assigned by KVM and you can't - for instance - run a CUDA simulation in the Linux host. You need to reboot X (you don't need to reboot the system).
The workaround/solution is to fire up KVM with GPU PCI passthrough to another guest (This time a Linux guest) and in there you will have full access to the GPU to do CUDA computations (or whatever else you want).