There are so many vulnerabilities in the baseband that it's not even funny. Even the QCOM secure boot process is full of holes. If a government agency wanted to drop a persistent baseband 'rootkit' on your device with full access to userspace, they could (unless you're using one of the few phones with separate userspace and baseband processors).
The DIAG commands are particularly fun. You can read and write memory on most phones. Some have locked it down to certain areas, but this varies wildly depending on manufacturer.