That issue can be dealt with using an IOMMU under host control. Those limit where the peripheral can DMA regardless of its firmware.
But more directly, worrying about one part having a backdoor is a lot better than worrying about twenty parts having a backdoor.