Humans are terrible drivers. If a self-driving car got into half as many accidents as the average human, it would save millions of lives. And kill people, to be sure, but fewer on net.
I also think you could make a reasonable argument that all cars should be banned, right now, based on how many people they kill, but since I don't think that's gonna happen...
I'm a pretty attentive and cautious driver. In the 20 years I have been driving I've been in one accident and that was because another driver was attempting to make an illegal left turn, came across two lanes of traffic, and t-boned me. So if self driving vehicles are only doing better than the worst human drivers, I'm going to be pretty hesitant to turn over control. I'd be in favor of that other driver that hit me using an autonomous vehicle though.
It doesn't seem impossible for there to be a long tail of driving (in)ability: most people drive pretty well, with a small fraction that are distractible or reckless enough to account for most accidents.