But the dirty little secret is that the baseband processor is still a completely uncontrolled subsystem, loaded with some propietary binary blob by trustworthy companies like Qualcomm. GPS and even the microphones are usually integrated into the baseband, not part of the application processor that runs your Android. So you have a perfectly capable ARM processor running a propietary RTOS system, written completely in C (or C++ occasionally) with access to all the vital periphery and a gigantic attack surface in dealing with all the mobile communication protocols. The only reason there hasn't been a complete breakdown yet is that it's difficult for amateur researchers to exploit, you need expensive RF hardware and the mobile communication protocols are huge bodies of closed committee designed standards. But it is without a doubt in the reach of the NSA, and they are probably actively exploiting baseband processors already.
(Interestingly, since baseband processor have grown in complexity, most smartphones can now update the firmware on them, so there are lots of firmware images floating around. I highly recommend just even running strings on them, its quite enlightening. Some examples from a Nexus 4 radio:
Failed do spoof USB cable disconnection
Assertion os_mutex_pool_ptr[mutex_index_in_pool].is_available == 0 failed
hsu_al_ser_open: hsu_al_ser_base_open for port NMEA (%d) returned failure
Conversion to UTF-16 failed! Returned %d, expected %d
Unexpected IP family %d - assuming IPv4
inflate 1.2.3 Copyright 1995-2005 Mark Adler
Received ARP Request
CxM - Received WLAN Early Grant Release
(Yes, these are format strings! And this device has all the good stuff: classic 2005 zlib, a homebrew network stack, homebrew character conversion routines, homebrew operating system, homebrew USB stack...)