But let's not measure AV safety against the average human driver, but against the most capable human driver.
Every time a self-driving car causes an accident in a situation where a human driver probably would not have (and sometimes in pretty obvious ways, like driving into a freeway divider), it will hurt acceptance of the new technology. Politicians will never miss an opportunity to grandstand against tech companies. People will be afraid to ride in autonomous vehicles.
Autonomous vehicles are dangerous in unpredictable ways. People might be injured or die in accidents that would not have happened with human drivers, as the result of software bugs rather than human decisions. When things like that happen, statistics about accident and death rates being lower with self-driving cars are beside the point.
Its politicians, who are all about "click-bait-emotionally driven" to bring them votes, you should worry about.
If you're concerned about the costs of a driver, I grant that the marginal costs per hour of driving is low, but the material costs seem high, unless a significant cost breakthrough is found in lidar production, or a technique breakthrough using multiple cameras; and the R&D costs seem pretty enormous -- people have been seriously working on this since the 80s and it's clearly closer than fusion, but it seems perenially 15 years away.
Everyone likes to think they're better than average (https://en.wikipedia.org/wiki/Illusory_superiority#Driving_a...).
I would rather keep the worst drivers off the road than keep the best.
A machine suddenly hitting a bug is whole new ball game though. I guess over time maybe you could build up a sense of where machines have trouble driving and adjust accordingly.
That isn't just a question of the safety of the autonomous systems though, it also includes consideration of which drivers end up using them. Some drunks will use auto taxis, lots won't.
I still think marketing it as a safety issue was the wrong approach this early in their development. Say "safety" and "cars" in the same sentence and people think of seat belts, air bags, and anti-lock brakes. All things to protect you from an accident on unpredictable situation.