Isn't this just an instance of "trusting trust?" How do you know the IOMMU hasn't been backdoored? "Open" firmware doesn't mean open RTL. Where is the line drawn?
But more directly, worrying about one part having a backdoor is a lot better than worrying about twenty parts having a backdoor.