The only solution that sounds plausible to me is a hybrid approach.

Much like when building a car you can make the engine as complex as you like - but you keep things simple with the brakes. Aircraft autopilots sometimes have three separate processors, programmed by three separate teams.

In a self-driving car, I'd expect a heavily audited core of code to take care of not hitting clearly visible stationary objects like concrete barriers and fire trucks even if other parts of the system relied heavily on ML.

> Or do you consider it fundamentally unsolveable?

I think self-driving vehicles will trade off some types of accidents for others.

For example, you see a running child who disappears behind a truck, then a ball rolls out into the road? A skilled, attentive driver would anticipate that the kid might run into the road. But that needs fairly detailed scene understanding. Accidents like that might increase.

On the other hand, a lot of accidents are due to lack of attention - and a machine can provide consistent vigilance. So accidents arising due to drowsiness and drivers fiddling with the radio could be expected to decrease.