That's a good point. If you see a pedestrian hit by a car anywhere near your car, you stop, right? If you see a pedestrian walk out into the road against the lights you at least slow down and/or cover the brake pedal because who knows what they're going to do?
I also think that this feedback loop of "Model the world, take action, observe, update model" is a key component of consciousness (whereas we model ourselves!) and key to AGI. I would not be surprised that AGI comes from a self-driving car development model.
The reality is that nearly all sensory input is novel in some way and a large part of the work the human brain does is quietly filtering out a massive amount of irrelevant input in a way that is computationally infeasible with current technology.
You might stop, yes. But many humans don't. That channel is mild and tends to be relatively work-safe; it doesn't take much searching youtube to find the rest.
It's hard to avoid thinking that maybe humans should never be allowed to drive automobiles.
I'm not sure that's actually true, but even if I posit it... "properly trained, well rested, and sober" does not actually describe humanity. You can revel in your moral superiority to the average driver, or you can make the roads safer.
AVs do not need to be better than the best human drivers. AVs need to be better than the average human driver. The average human driver is shit.
Like the fact that a cyclist on the other side of the road is a hazard to you, not because of the cyclist, but because an impatient driver stuck behind them is likely to pull out suddenly round the cyclist, forcing you to brake.