That would be the case if people are rational. However, people are not rational, and don't like not being in control, even if the chances of them dying are far lower. It may be safer to share the road with self-driving cars, but it's still scarier to rely on a machine that you don't understand.
The most important barriers to adoption are social, not technological, and that means that even one major catastrophe can cause a wave of backlash, regardless of whether or not it's justified.
And that's why it needs to be rock solid.