Which FLOSS operating system are you running on the baseband?
I didn't like the samsung-esque install of their base, so I recompiled to what I liked.
Baseband processor is separate core that runs its own, realtime OS, that handles radio communication. These OS are not opensource.
The application processor talks to baseband via RIL (rild in Android), which is basically a form of IPC via shared memory.
To recapitulate: In your phone, there is another CPU, with closed-source OS full of bugs, connected via air to network and it has full access to RAM of your application processor.
I hope that it's obvious, that it does not matter, what OS runs on the application CPU.