You'll find the modem on most smartphones is connected via USB - or rather its chip-to-chip version, HSIC. For SoCs where it's on-die - on the same bus/fabric - they will (if it's not an idiotic design) use an IOMMU of some sort, to prevent DMA from having access outside of its sandbox.
Even if it's a modem on the other end of USB - which will almost certainly be using DMA, but at least host-programmed DMA - that's no guarantee. Google "Evil USB". USB is a overly complex stack which has and will continue to result in countless vulnerabilities, regardless of what you do with it.