The fragility of AI decision-making in general absolutely is an issue. Feel free to post a submission when it has been solved to a level that matches or exceeds human capabilities.
This is not unlike the human body's deeply-evolved reflexes - before the neocortex can even process that a pan is boiling hot, the nervous system bypasses the brain and and tells the hands to drop it.
In the case of driverless cars, DannyBee is absolutely correct: truly driverless cars will not make decisions from a single data point. I have some past experience with unmanned aerial vehicles and those things had insane amounts of redundancy in almost every signal and decision making process; driverless cars are more complex and will come decades later and will probably take this even further.
In the case of driving, however, the problem of deciding what behavior to veto is itself a hard problem - perhaps the hardest of all. When the system monitoring the AI is itself AI, the argument becomes circular - which is not to say it cannot be solved, but I think it makes it harder to dismiss the concerns.