Proxmox has SPICE which is what I use for VDI instances on my server - it's not as good as native but much better than VNC (in my eyes it performs similarly to RDP and is not as fast as Citrix or Horizon). I don't think it supports Mac guests though, only Windows and Linux.

I've never used it, but I guarantee that Proxmox can pass through PCIe devices with VFIO, and USB requires no hardware support at all.

vmkernel hardware passthrough has exactly the same requirements (IOMMU support).