Even if they do cause rare accidents they ll be successful. I think in the end humans will adjust to drive more rationally when they learn how to spot autonomous cars.
It's worse on county roads (like in my local area, 50mph) without passing lanes. If you drive the speed limit you can almost guarantee to be passed by many cars, sometimes even dangerously, because it annoys people to go below the speed limit they perceive as correct.
I like to go around the speed limit (admittedly, simply to avoid tickets), usually ~3mph over, but it will annoy me to no end if i have to drive 5mph slower because of someone in front of me. On these popular county roads i speak of, i usually go faster than i want (5-10mph over), because i don't like being dangerously and needlessly passed.
Now, i'm not saying the self-driving cars are wrong in any way. I'm also aware that the roads i speak of will be among the last to get self driving cars. Nevertheless, i think humans will have to learn to drive far less aggressively. Though, when self-driving outnumber the normal drivers, it may not matter.
Granted, Google's security is about as good as it gets for most stuff, but what about companies like Audi, BMW [1], Daimler, Ford...Toyota [2]? Many people will still buy cars from them well into the future.