In my research on phones from the Unrevoked project (admittedly, 4+ years ago), this was the case: the baseband and the CPU shared the same memory. The baseband memory was carved out from the CPU such that the CPU could not access it, but the microcontrollers serving the baseband had CPU access, as I recall from the Qualcomm boot documentation: the chain of trust from CPU boot was established by the baseband processor, not the other way around.

I would imagine that things have changed a little bit, but the baseband back then, and I imagine still now, is considered to be the ultimately trusted element of the system. I'd be surprised to hear that they've changed so much that the baseband doesn't still have full control over its host system.