There's literally no such thing as "known to be safe" on a road, a crash could happen right in front of any of these cars, the car promptly kills its occupants through a mistake and then that's the end of self-driving cars for a decade.
But it looks like we're wrong. First of all, there have been crashes with even Tesla's autopilot, and it didn't seem to affect anything.
For another, this same logic would've probably meant that driving itself would never have become popular in the first place, or would've been set back many decades because of car crashes. That doesn't seem to have been the case.
I'm not saying these things can't kill an industry, but I think people are pretty tolerant of even unthinkable dangers beforehand, if the incentive is high enough.
Though possibly a bigger backlash could come when a self driving car kills someone in a situation where there are only 2 bad options to choose from. Being that with a human driver nothing was pre-planned people move on. But with a self driving car where such situations have to be decided in advance the backlash could be much more significant.
Consider the alternative: autonomous cars theoretically get no production use and then suddenly go "on sale" for any buyer to use them in whatever route or condition they want. This would result in an extremely chaotic situation where suddenly there is no opportunity for gradual production deployment to slowly find the edges of their capabilities (which in some cases could result in accidents) -- instead the whole buyer-base of autonomous cars would be exposed to those edge cases, in full, from the moment the cars became available.
edit: Also I'm not sure what you are asking about direct knowledge: I'm stating observations of reality that self driving cars are already being rolled out incrementally in places/routes that they are best suited to.
People driving have accidents all the time right now that could have been avoided if proper automation was in charge.
At some point if you have an accident and kill someone, and it can be shown in court that if you'd been using the easily available technology that person would still be alive, people will start being prosecuted for gross negligence or worse.
Surely, if you only bought a more expensive vehicle, you could have saved a life! /s