This is Tesla, and Comma argument. I think it is a fallacy.
Yes, we can drive with just a pair of eyeballs. But you have to realize that the entire road infrastructure is made for humans. Signs are made to be easily recognizable by humans, human reaction time is taken into account, we have a good idea on what is easy to understand and interpret by humans, but we don't give a fuck about bots when building roads. It means that we start with an unfair advantage. And we are asking bots to be better than we are in that environment.
Humans have a remarkable ability to contextualize a situation. For example, if we see someone walking on a side lane, marked for pedestrians. We know that that person is not a threat, he is unlikely to leave his line and therefore we can pass at a normal speed. That's unless we see that the person is visibly drunk, or maybe he is a small child, in these cases, we will slow down, because he may suddenly jump in front of the car. Bots probably won't know that, and usually, for safety, they will always slow down. Similar situations happen all the time, and that would make for a terrible ride, so what should we do? Make the system ignore the "drunk man" or "small child" case and possibly kill people?
Also one shouldn't forget the communication aspect of driving. Drivers have all sorts of way of showing their intentions: turn signals obviously, but also subtle movement, you may go forward a bit to signal you intend to merge into traffic for instance, wait a bit to let someone know he can pass, etc... New drivers often struggle with that, which may cause more experienced drivers to misread their intention and potentially cause an accident, unless these drivers notice that he is indeed a newbie... or a bot.
One answer is for bots to compensate their weakness with things they are good at. Lightning fast reflexes is a given, but sensors are another aspect. Unlike humans, bots are not limited to a narrow field of vision and visible light. It is an advantage they should use. We humans are insanely good at reconstructing a 3D image from our 2D eyes, a good part of our brain is dedicated to that, computers can't compete, unless the computer is assisted by something like a lidar.