I don't believe this is entirely true anymore. Yes, years ago, the baseband processor (and firmware) had full DMA capabilities to the RAM ostensibly managed by the OS, and could do nefarious things if it wanted to. But I believe nowadays the baseband is a bit more isolated, and communication with it is mediated by the CPU and OS.
Some manufacturers likely still implement the "old" architecture, though.