Everything about it is fickle. Motherboards and USB controllers present themselves different from maker to maker, and SKU to SKU. Any change in your hardware -- including plugging in something like a USB device -- sets off a chain of fighting the host OS and virtual machines for who gets to use that device, who gets to see that device at all, and the drivers needed for the hardware.
No one is making user software with VFIO in mind. You will get random apps that do not launch in your virtual machine despite everything else running smoothly.
You're screwing with it near daily, certainly weekly. Need to update a driver, such as your GPU on the virtual machine? May god have mercy on your soul that everything works after, or that you can even get the host OS to deal with the GPU correctly at all.
I can sort of tell you haven't really read up on one of these setups because you seem incredulous that running Windows inside Linux is somehow finicky.
It's not ready for prime time, period.