However, we don't know the risk/reward associated with ceding control over our safety and the safety of others to an internet controlled computer. Over time, human controls (and skills) atrophy, so that when another Carrington event happens humans will not be able to recover. It also makes us even more susceptible to hacking attacks - wide-spread use of autonomous vehicles will mean that a sufficiently skilled hacker can kill you from afar. If not with your car, with someone else's.
So, no, I'm not particularly excited giving computers control over my car, my home, my life, even if it seems like a good idea. The downsides are just too substantial, and honestly humans can and should retain some responsibility for their own well-being, on first principle.
So we can remove the physical buttons when we remove the steering wheel, but not before then.
I agree with your overall premise, but this is a strange conclusion. If human drivers are the problem and they cannot be trusted to drive without distractions, it would be far easier and cheaper to fix that by investing in public transit.