One reason I've heard is that when you're on the phone with someone, they don't know what's going on in the car whereas if you're talking with the person next to you and you suddenly need to focus all of your attention on the road, they can stop talking rather than adding to your distraction. Plus they provide a second set of eyes. I don't know how accurate and reliable that theory is, but it makes sense to me.

I also wonder if the mental effort of listening to a voice coming through on a speaker would make it harder to focus. I tend to have trouble hearing people when I'm talking on the phone; but that might just be me.