You chose to make your point with probability, which is actually what will most likely make people accept driverless cars in the end.
It's amusing to read the same reasoning after striking away "driverless":
"You are far more optimistic than me about the future of cars. Given N cars, some fraction p will eventually run over and kill small children. Stand back and watch the hysteria and the end of cars."
or with elevators:
"You are far more optimistic than me about the future of automatic elevators. Given N elevators, some fraction p will eventually malfunction and kill small children. Stand back and watch the hysteria and the end of elevators."
If the accident probabilities can be brought to acceptable levels, why wouldn't we adopt them?
PS: With trains and subways we have 1 driver per dozens of passengers, where the savings gained from reducing it to 0 may not outweigh the risk increase/loss of safety feeling.