...but near term "If my child died in a situation where I know without a doubt an AI caused the accident, it would matter a hell of a lot to me."

There are going to be a lot of lawsuits that will sort out the legal frameworks after the first wave of failures by automated conveyances....

Absolutely, you're right. But as a singular entity, I'm a human, and an AI is "other". I know I perform better in 2 cases, and AI statistically perform better in 100 others, but I believe I'm more special than statistics in those 100 other cases, so I only care about the cases where I could have made a saving difference.

I guess I'm suggesting there's bound to be a (n irrational) paradox where everyone wants other drivers to be replaced by AI, so they themselves can drive safely and avoid the edge case accidents.

That’s all in a hypothetical future in which autonomous vehicles work as intended, are well tested, and are better than people. We’re still nowhere near that kind of lvl 5 automation, and just testing on open roads.

Be careful of the utopia trap: “anything is acceptable now for the rosy future imagined.”

Even if the cumulative amounts of fatal accidents is reduced, certain types of accidents happen less, if different types of accidents happen more often, say accelerating into a wall uncontrollably, that is not optimal. I worry that we will accept 'slightly better' as good enough, along with the consequences that come with it.

At some point, economics and actuaries come into play. It will be worrying when 'AI is safer than humans!' can be pointed to as a reason to not correct expensive, perhaps intractable, AI failures.

> I'm sure that a large portion of people would still be really irrational about it and that worries me a lot

I used to think that. The Tesla AutoPilot accidents have inspired no such reaction, though. The moment might come more when a self-driving car is used by a terrorist or school shooter as a getaway vehicle than because of someone getting mowed down by an errant algorithm, the latter being a cause of death we've been relatively numbed to.

Or a self driving bus with children could get confused by a weird sticker and jump of a bridge, hit a giant truck stopped in the middle of the road.

We are speculating here, we need numbers to see if this self driving cars are better, how do we get this numbers though? Is it fair for the citizens of a city to be forced to be part of this tests? Are we sure the correct number of incidents are reported from this tests? are the tests good enough for real world (are they testing in all conditions are avoid some streets or some weather to keep the numbers looking good)

I would like to add, if the vehicles entirely depend on their AI, then there should be at least 3 or 4 compute nodes and a voting system. Should a node fail or have false information, the other nodes should override.

Them competing to be the safest means that none of them are as safe as they could be working together, though.

If every company has their own secret suite of test cases then different companies can specialize in different aspects of safety, and different AIs will be tuned to watch for different conditions.

Imagine if instead of that they all worked together to define a rigorous test suite. Then they would all be striving to excel against all of the tests that the best of them could come up with. Wouldn't that result in more rigorous testing than any individual company would do? Especially if the results of all of the tests were public?

To go another step further, imagine an open carAI platform that had the aforementioned test suite and a full simulation platform for testing changes, with different car manufacturers represented on a committee that oversees the carAI platform. Separate the smarts from the base car a bit and have some sort of abstraction layer between the smart bits and the car bits. As long as the abstraction layer is configured properly then different AIs would be interchangeable/upgradeable on the same base hardware. All car companies (and tech companies, and interested individuals) could collaborate on building the best, most efficient, safest car AI possible. People on older hardware would get all the same safety improvements as people on newer hardware (though hardware improvements would obviously improve things like sensor quality and quantity and the like), there wouldn't be fragmentation between ai ecosystems with poorer people trapped on older releases with lower safety standards while the rich get the latests and greatest and safest cars, etc.

Obviously competition is better than nothing, but is it really better than an open, collaborative alternative?

The market doesn't bear this out with current cars, so why should autonomous cars be any different? Cars are marketed as powerful, sexy, vehicles that imbue with strength to traverse the harshest conditions in the worst weather. Macho vehicles. The only car maker that seems to have a sterling reputation for safety is Volvo, and they're a niche manufacturer.

>I really really doubt that! The car manufacturers will fight tooth and nail to be seen as the company providing the safest cars

We seen competition not working in many sectors, like ISPs is US, operating systems for desktops or mobile phones...

IF the self driving component in the car could be swapped so you can buy say Tesla but get the self driving package from BMW or from Google then that could help competition, but without this if all companies have a similar failure rate then they can concentrate on competing on marketing,horse power, price, efficiency and not adding some extra safety, at least until some important person or a big number or children get killed by a very stupid issue like a car hit a wall because it did not see it because it got confused by some drawings on the wall.