The trouble is when we swap "human error" for "machine error". Because humans screw up all the time but a machine is going to fail in a way that'll likely be very obvious to us. That is going to be take an emotional toll, I think, and will be a hurdle to get over for self driving cars.
No one wants to be told "sorry but your family died because of a bug in the car's software told it to make a left into a semi".