The baseband processor may have unrestricted access to the entire address space of the device or to address space which the application processor (and the operating system it's running) implicitly trusts.
AFAIK, access to the baseband and vice versa is through a network inside the phone. Physically, these two computers are separated, and communicate only through a well-defined network interface. No poking in other computer's memory.