Aside from being vulnerable to malicious actors, self-driving cars make "cold-hearted" mistakes that don't make sense; mistakes that humans would never make. Sure, the total number of mistakes they make will be lower than the equivalent number of accidents caused by humans. However, having human life lost to the type of mistakes self-driven cars make is, in my opinion, an unacceptable price to pay.
If I must loose my life to a car-related indecent, I want the it to be caused by a human, and not some quirk or edge-case in an algorithm or neural net.