That still ties it to a physical VM host doesn't it - what happens if you want to host the VM in a cluster and have it migrate?

You don't even need VT-d for USB devices, only PCI one.