One of the problems is insurance. Right now, everyone needs to have insurance to drive. If I accidentally hit a pedestrian or rear-end another driver they can claim on my insurance.
Let's say that a software bug results in someone getting hit and injured by a self-driving car. Who's liable? I am not sure that this is FUD, so much as a genuine question. It is because driving is risky that drivers must have insurance. Presumably most people would want self-driving cars to be insured if there was a chance of failure (even planes, one of the safest forms of transport, get insured against crashing).
We could very well have a situation where cars self drive, but drivers are still expected to pay attention to the road and 'drive' to avoid liability.
I suspect there's going to be a lot of lobbying on both sides before mass deployment of self driving cars.