Cars that drive themselves 99% of the time but 1% of the time give up and hand off to human drivers are going to kill the drivers or somebody else about 1% of the time.
And to build a self driving system that can give you a reasonable time to actually assess the situation and respond to it? That's the same system that's 99.9999% reliable.
This is the crux of why self driving cars will simply not work in the foreseeable future unless sequestered to their own tightly controlled road networks.
I don't think the problem will be any situation with short reaction times at all. Nothing at freeway speeds, or situations like the Uber accident. That I think is where autonomous cars will shine, because sensors never get tired and reaction times are great.
The weird things that will happen which I count to the "not going to solve any time soon" category will be when the car comes to a completely snowed over roadworks, in the middle of the night, with the diversion signs completely hidden in snow. Construction workers barely visible in the snowstorm. Are those guys roadworkers or pedestrians? Are they working? Can I pass here? Will I get oncoming traffic because they narrowed it to one lane?
When things this weird happens at highway speeds or anywhere else where reaction is important - humans probably fail too. And at that point it's not really a question of technology but one of trust. Can we allow autonomous car to kill tons of people every year, with the sole excuse that humans would have killed all those people too, and then some? I'm not convinced of that either - I'm only arguing that from a technological standpoint, it should be possible to reach the 99% cars within a rather short timeframe. Those cars may be left on the scrapheap of history because of legal or ethical reasons, however.
In reality it's mostly just the AI expected one thing, and observed another - so something's not working right and it seeks a disengagement. California requires companies to quantify disengagements and most go a step further and specify the reason for the disengagement. I think the reason for this is precisely because of your intuition -- thinking that disengagement means imminent danger. Even for companies with relatively large numbers of disengagements, there were generally 0 that involved any danger whatsoever.
These would be (remote) car pilots, probably specialized in specific areas.
The auto-land disconnect scenario isn't applicable to cars. It happens because jet liners are _flying_ and suddenly ceasing to fly in a jet liner is both very bad and perhaps unavoidable in the absence of enough information to operate the plane within parameters.
In contrast when a car becomes uncertain about what to do it's not flying so _stopping_ is almost always a good choice. It's not ideal, it may block traffic and be a nuisance, it might even cause a small accident of some sort - but it's very likely to end with everybody walking away, not with a burning wreck and dozens of dead.
https://blog.piekniewski.info/2017/05/11/a-car-safety-myths-...
> Now it is important to note that the definition of a "disengagement event" may vary between companies. Most companies report every case in which a human grabs the wheel for any reason. Waymo (*) only reports the events, in which if not for the human intervention the car would actually cause a dangerous situation [read more here]. The way they do it, is for every physical disengagement they gather all the sensor data and next simulate multiple scenarios. If these scenarios lead to a dangerous situation, such event is being reported. According to Waymo in 2016 nine events would have lead to the car hitting an obstacle or another road user, approximately 1/10 of all disengagements they've reported (124). Hence there is such a gap between Waymo and the rest of the pack.
I also think it's hard to generalize from Waymo employees being attentive and general road users achieving the same thing.