Why do you disagree with hands-free devices? What do you think about conversing with a passenger during driving?
I also wonder if the mental effort of listening to a voice coming through on a speaker would make it harder to focus. I tend to have trouble hearing people when I'm talking on the phone; but that might just be me.