This is why I do absolutely nothing on a new phone until it's been unlocked, rooted, wiped and Cyanogenmod-ed.
Unless you tunnel all your data traffic they'll also get copies of that and may sell pseudo-anonymous website usage statistics to one of the web metrics businesses. In some countries they also intercept and modify web content that you might view.
And what can you do about the baseband radio processor and its code? Nothing. Assuming you have a tunnel in operation, the phone could still be collecting and quietly sharing metadata and you'd never know.
The baseband has its own processor and, to my understanding, pretty much complete hardware control of the phone.
I don't deny that that the benefit is non-zero, but I think saying "it's not a silver bullet" is still overselling it. I think it would be more accurate to describe it as "It's all I can do, and it's better than nothing."