Unfortunately AMD video cards don't really support PCI passthrough- I'm in the same boat as OP with an RX 5700. The issue has been present for several years, and is somewhat fixed by community patches, which makes me suspect that this is being intentionally overlooked by AMD so as to not allow their consumer video cards to be used in the data center.