Let’s say autonomous cars, once fully implemented, are even 10% safer than existing drivers. And testing in real-world conditions, speeds up the implementation by even just a year. We would need about 3k deaths at the hands of the autonomous drivers before the trade off was no longer worth it. This is only considering the US. And based on very conservative estimates of 10% improvement and a reduction of only a year savings in time to deploy. The more likely scenario will put that number in the 10s of thousands in my opinion.
Personally, I would much rather we slowly develop and deploy these cars in the wild rather than continue testing in some controlled environments. At least this way we can monitor progress and judge their readiness as a society. Plus, the sooner they happen, the better for everyone.