Wouldn't SR-IOV support (which is rumored in nvidia's chips) make vfio on this kind of setup possible?
Even if it's possible to implement proper VFIO for Nvidia GPUs, I expect them to fight such an implementation with tooth and nail in their drivers. This is the same company that artificially limits the amount of transcoding streams in their drivers, mind.