Beyond experience and fear, I think an autonomous vehicle will need a reasonably accurate mental model of how other human drivers on the road are likely to react.
> Situational awareness or situation awareness (SA) is the perception of environmental elements and events with respect to time or space, the comprehension of their meaning, and the projection of their future status
There's an bearish case to be made - not only are the state-of-the-art algorithms nowhere near this - it takes kids quite some time to acquire this skill. Playing ball games with my kids, I've noticed even a 3yo still doesn't intuitively grasp Newtonian mechanics - they're trying to catch the ball where it is now, not where it's going to be in a few seconds. 8yo - notably better, though not fine tuned yet. Heck, I know some adults who're lacking in this respect.
I've noticed over the years that the morning commute is a symphony of madness. Drivers going over the speed limit. Drivers doing illegial things in order to get to work, and keep the traffic moving.
Cops seem to let a lot of infractions go during the morning commute. (The rest of the time they let nothing go, but in the morning commute they just seem to know it's a symphony of driving?)
I picture a self driving car going 25 mph with 20 vechicles behind it.
I don't think you could legally program a car to break any law, even in emergencies. Well you could, but lawyers would have a fun day in court?
It's as simple as that.