Like, I'm cruising along at 45mph and a plastic bag is blown in front of my car. Yes, I see an obstacle and need to make a decision about what to do - but I do this as a human who knows what a plastic bag is, and how it is far safer to drive right over it than slam on the brakes. I know a plastic bag is not, say, a Pomeranian.
Another example: I'm driving, and from beyond my plane of vision, a basketball bounces onto the road up ahead. I can't see past a building, but I know I'm near a park where kids frequently play, and I know it's possible a kid comes running into the road to grab the ball. I proactively slow down. I guess you could say a leavel-5 autonomous car would see the ball and slow down anyway, but there's still that lack of cognition that concerns me.
We like to think of driving as a purely rule-based game that is simple enough to model and train against. And I believe in 99% of situations, it is. But in that 1%... the plastic bags and all that...