But it doesn't have to be perfect, just has better than human drivers. That doesn't seem too difficult.
People are conditioned to accept that we will kill each other with cars every once and a while. I'm skeptical we will get to the point any time soon where the general public hears about a family killed by a self driving car and just shrugs it off. There will be intense pressure to get them off the road.
It may not even be possible to solve without something radical like banning human drivers or inserting electronic nodes directly into our roads and infrastructure to aid autonomous vehicles. The existence of human drivers might make the problem simply impossible to solve in a way acceptable to society.
Software professionals are somewhat notorious for underestimating the difficulty of their projects. There's a massive amount of literature about that, proposing various techniques of mitigating this on a personal, team, or organization level.
That being said, I think in this particular case the problem was more of an overestimation of what the ability to work with highly dimensional data (as in ML algorithms) can give you (and underestimating the practical problems of deploying the ML-based systems). Basically, people tried running 30+ years old algorithms on GPUs while feeding them the ungodly amounts of data and realized that, with sufficient horse-power, they can make them (finally) work (as in: do something genuinely useful). This built up a lot of hype, of which self-driving cars are just one offshoot, I think.
Anyway, (if it's not evident from the above ;)) I find your arguments convincing and share your doubts. Self-driving cars are probably not impossible to achieve, but to get there we either need decades of research and progress or a couple of very high-profile breakthroughs in tech and theory. I won't hold my breath for neither :)