It's already illegal to use a cell phone while driving in Ontario, Canada. Hands-free devices are still okay, however (I disagree with that part).
I also wonder if the mental effort of listening to a voice coming through on a speaker would make it harder to focus. I tend to have trouble hearing people when I'm talking on the phone; but that might just be me.