Second, it matters if those possibly-hostile cores have network access. Technically they could subvert the software on the main CPUs to communicate with command/control, but that seems like raising the bar to such an attack. Whereas with the standard mobile architecture, I can totally see some phone manufacturer getting the "bright idea" to have the baseband processor collect statistics on the application processor's software for market research.
But I agree with your general point.
Still, I think moving in the direction of Linux phones gives us a starting point to do something about this insecurity - proving the market allows there to be devices that truly separate out the cell modem.