This is my line of thought as well. Same problem I have with the anti-nuclear environmental crowd. Are there substantial drawbacks? Absolutely.
That doesn't matter. The important question is not "Does this new technology solve all previous problems?" but rather "Does this technology solve a net positive number of problems while offering more opportunities for improvement?"
With decades of empirical data that average humans are pretty bad at reliably manually piloting powerful, heavy vehicles, I can't see a reasonable reason to reject deploying autopilot technology in an aggressive manner. And that sadly means a few deaths. But those should be measured against the number that would have died were nothing done, not against zero.