I don't believe people are using their full AGI when driving (and the full "AGI" may as well happen to be a set of basic pattern matching capabilities which we haven't discovered yet). After decades of driving the behavior is pretty automatic, and when presented with complex situation following a simple rule, like just brake, is frequently the best, or close to it, response.
As things stand emergency vehicles have to cope with a reasonable minority of people who completely panic and actually impede their progress.
Or hacked maliciously.
An incomplete list includes:
1) Integrating all the camera views into one 3-D vector space before training the neural network(s).
2) A large in-house group (~1000 people) doing manually labeling of objects in that vector space, not on each camera.
3) Training neural networks for labeling objects.
4) Finding edge cases where the autocar failed (example is when it loses track of a vehicle in front of it when the autocar's view is obscured by a flurry of snow knocked off the roof of the car in front of it), and then querying the large fleet of cars on the road to get back thousands of similar situations to help training.
5) Overlaying multiple views of the world from many cars to get a better vector space mapping of intersections, parking lots, etc
6) New custom build hardware for high speed training of neural nets.
7) Simulations to train rarely encountered situations, like you describe, or very difficult to label situations (like a plaza with 100 people in it or a road in an Indian city).
8) Matching 3-D simulations to what the cars cameras would see using many software techniques.
Yeah, a Tesla couldn't possibly drive into a stationary, clearly visible fire engine or concrete barrier, on a dry day, in direct sunlight.
I dislike saying anything in defense of tesla's self-driving research, but let's be accurate.
At least, I never would...
A million people are killed globally each year by motor vehicles. Staggering amounts of pain and injuries. Massive amounts of property damage. Tesla's cars are not supposed to be left to drive themselves. The chance to save so much carnage seems worth letting some people driving Tesla's, that fail to pay attention to the road, suffer the consequences of poor decisions.
Plus these problems are likely too be mostly fixed due to the fact that they happened.
Start-up religion doesn't really work when there are lives on the line. That's fine for your social media platform du jour but please don't bring that attitude to anything that has 'mission critical' in the description. That includes medicine, finance, machine control, traffic automation, utilities and so on.
Not to say that we shouldn't be cautious here, but over-caution kills people too