"Isolation" of driver code that can talk to on-SoC hardware is just not very meaningful. You can only have real driver isolation if it's enforced on the hardware side via some IOMMU mechanism (or by keeping the hardware isolated on the USB bus, etc.), otherwise you're just adding pointless overhead for no real benefit.