Comment by OnlineGladiator | Hacker News Reader

Comment by OnlineGladiator | Hacker News Reader

OnlineGladiator

> Waymo says its vehicles were involved in 47 “contact events” with other road users, including other vehicles, pedestrians, and cyclists. Eighteen of these events occurred in real life, while 29 were in simulation.

If they drove 6,100,000 real miles and ~5,000,000,000 simulation miles, and had 18 real contacts and ~29 simulated contacts - that strongly indicates their simulation is not representative of reality.

Edit: My (old, now changed) numbers were wrong but my theory was right!

So they drove 6,100,000 real miles and had 18 real contacts. Between June 2019 and April 2020 they drove ~5,000,000,000 simulated miles and had ~29 simulated contacts.

So their simulator is 3 orders of magnitude safer than the real world. The only number that's really fuzzy here is 29 - the number of simulated contacts (because the dates don't match up perfectly, as this report from Waymo started counting in January 2019). But no matter how you slice it, there is clearly no way the simulated miles are representative of reality.

https://techcrunch.com/2019/07/10/waymo-has-now-driven-10-bi...

https://blog.waymo.com/2020/04/off-road-but-not-offline--sim...

The 6.1M miles was self-driving in real life, with a safety driver in the car.

The simulation portion only comes into play if the safety driver intervenes in a real life situation. In this case, they go back and simulate what would have happened in the safety driver did nothing.

> that strongly indicates their simulation is not representative of reality.

If they take the common software testing approach where every time they discover a fault, they add a simulation that reproduces it alongside the code change that fixes it, we would expect the simulation to contain a great many scenarios where collisions don't occur.

Indeed, I would be more worried if they'd resorted to real-world testing before they'd solved the known problems their simulation had revealed.

Of course, whether quoting a number of simulated miles driven is relevant to anything is another question....

That's a good point. It still suggests there is no apples to apples comparison between the real world and simulation though, and that those numbers can't really be considered equivalent or even compared in a useful way.

If you're saying the simulation should've revealed more problems than the real world, wouldn't that mean there'd be more contacts through simulation (because that's where you discover them) and then you'd hope the fix exists before rolling it out to the real world, so you'd expect to have less contacts there? What they're reporting is the opposite of that though.

Obviously I don't have enough information to draw a scientific conclusion here - I'm just pointing out there is an enormous discrepancy between their simulated miles and their real miles.

That's not what the article says. They drove 6.1 million miles autonomous with a safety driver.

Where did you get those numbers from? according to the article 6.1M miles was driven on the roads with safety drivers, and additional 65000 miles were driven without safety drivers.

"Simulated contacts" are when safety drivers engages, and Waymo investigates what would happen if they did not, so this is as close to the real life as it gets.

The simulated contacts are when Waymo investigates what happens after a driver disengages and concludes they would have had a collision, not the actual contacts measured from simulation.

I know, confusing right?

Reply on news.ycombinator.com