If there is anything I've learned in my career it is that building systems that are 100% completely error-free is pie in the sky thinking even in moderately controlled environments. Suffice it to say, I will be a very late adopter although I think the efforts are very exciting.
We don't really think about it like this, but things like anti-lock brakes and automatic transmissions are all on this same continuum towards "driverless" cars.
A next step might be something like adaptive cruise control: your CC is set at 60 but the car ahead of you slows rapidly and your car automatically applies the brakes.
We already have that.
I would say it is similar in essence to air travel. Even though commercial airline travel is way safer than driving, I'm pretty sure more people fear airline travel due to the element of having no control. Unfortunately when they release probably any automated car crash, as rare as it may be, will be played up as death by machine.
I think they will have to really play up the ability to easily override the automation (even though I doubt it will mitigate automated car crashes in that once you get used to it being successful you'll probably be reading a book when the override is actually needed)
Old person hits gas instead of break, crashes into robot car. What would a human do differently? teenager runs red light, gets tboned by robot car. What would a human do differently? Child runs in front of car. What would a human do differently?
It might take a few years, but thousands of hours of video of humans making avoidable mistakes will probably change people's minds.
The most important barriers to adoption are social, not technological, and that means that even one major catastrophe can cause a wave of backlash, regardless of whether or not it's justified.
And that's why it needs to be rock solid.
I think—and hope—we'll be surprised at how quickly we acclimate to magic. Sure, we'll be completely freaked out by letting go of the steering wheel. Then we'll be irritated at how our car thought the leaves blowing across the road were an obstacle. Finally, we'll wonder how we ever survived our own driving and we'll mourn for those that didn't.
With the amount of data required for self-driving cars, the entire situation could be recorded - things external to the car. It would be like the airplane black box, but for all the vehicles in close proximity too.