Hacker News Reader

Comment by maliker | Hacker News Reader

That's around 7.71 crashes/million miles driven by Waymo.

By comparison, US national statistics from 2018 [1] are around 2.08 crashes/million miles driven.

Seems like they're far from human performance. There are ton of ways to slice this data, though. I used national stats for accidents where the police showed up; maybe Waymo's data includes no-police accidents. It would also be interesting to compare injury statistics. And of course this is Phoenix-specific data from Waymo, which is a place with good road conditions.

[1] https://cdan.nhtsa.gov/tsftables/National%20Statistics.pdf

(edited to include simulated crashes, which were only averted through human safety driver intervention.)

View on HN

I thought your Your 2.08 crashes/million miles number had to be wrong, since if that were true the average person would have about 1 accident every 50 years or so (which doesn't match my experience of everyone I know having accidents).

Your number was "police reported accidents" which was apparently 6.7M. Anecdotally, I have been involved in 10 accidents or so as a passenger or driver, and none were reported to the police.

So I can't find good numbers on what the real number is but I think its conservative to estimate it at 1/2 are reported to the police. In addition, "Crashes" mostly involve 2 cars, so the "police reported cars that crashed" is roughly 2x, or >12M.

Basically, those changes makes it roughly 8 crashes per million miles for normal driving, or Waymo being basically the same.

I still think that accident rate is too low though. This estimate is 6 crashes every million miles based on allstate data https://mashable.com/2012/08/07/google-driverless-cars-safer....

If that was the case, then humans would be at about 12 vehicle crashes/million miles, almost 2x as dangerous as Waymo.

So about 2x better than human performance?

Great analysis and new data!

Thank you for getting it started. I'm not 100% sure I understand it all. It's obviously important to figure out if it is safer or riskier.

>> So about 2x better than human performance?

If you really want the comparison to be on an even footing, you should consider only the crashes that happen during miles driven by humans in Phoenix. Not all over the US.

In Phoenix only- because Waymo would have many more crashes outside Phoenix (e.g. New York or LA, from what I'm told of the traffic conditions there) and because so would humans.

This is an excellent point, it is the same problem as trying to compare autopilot crashes to human crashes. The context is critical.

I think even then, we do not know enough to make any such claim. When did the Waymo cars drive, during the day, in rush hour, at night??? Did they simulate the typical use of an average car or just drove around in the next suburb.

And yeah, only Phoenix, so I assume few rainy days, barely any mist, snow, freezing and so on. The info of driven miles is nice to have but does not translate to normal human driven miles

Phonenix doesn't get much snow / ice / crazy fog / crazy rain. The roads are wide. The amount of traffic is low compared to, say, NYC or LA.

So yeah, they are not near human level yet.

I think there's a solid chance that a 'near human capable' driving system will handle slippery roads better than the human average. The system will be in a modern vehicle with traction control and likely make conservative judgments about road condition.

There might also be differences in how accurately a human or machine can model which maneuvers can/can't be done without losing traction. I could see this going either way. Maybe a human will be able to identify spots (like puddles) where traction will be especially bad, but a machine can do a detailed physics simulation of a maneuver before trying it.

In whatever way a machine does adjust its behavior for slippery roads, it will probably be more consistent at it than a human. Humans have to deal with force of habit, so for example they may momentarily forget that they can't brake hard or they shouldn't take that turn at a familiar intersection at the same speed as yesterday.

also poor visibility. humans need to rely on their eyes, a car with a bunch of lidar sensors is absolutely going to outperform us in fog, blowing snow, or heavy rain.

I suspect that Waymo will do better compared to a human outside of Phoenix than inside it.

The hardest thing that Waymo will have to do is to interact with humans behaving oddly. Phoenix has those, just not as many as NYC/LA/SF. But one thing computers do exceedingly well is scale. Dealing with one strange human is very hard for a computer, but dealing with two or more of them is very similar.

Snow / ice / rain on the roads means changing underlying models, behaving more cautiously. Computers can switch modalities much quicker/easier than humans.

Fog and other visibility problems are sensor issues. Humans are stuck with the old Mark 1 eyeball, but self driving cars have a mix of sensors. These sensors may currently be worse than eyeballs in fog, but may not be in the future.

Even given those advantages, you still start in an easy location. Going from 0 to 1 is really hard, give yourself all the advantages you can. But going from 1 to N is a lot easier.

Fwiw your conclusion is not really supported by the rest of your statement. Just because scaling from 1 to n is easy doesn't make scaling from zero to 1 feasible. And even if it is (feasible) there's no guarantee they'll be able to do it soon.

I'm having trouble figuring out how you're disagreeing with me, it sounds like you agree with me, that 0 to 1 is the hardest part.

Maybe a stupid question, I don't know your exact calculations, but wouldn't you be half-counting the number of collisions, since a collision is almost always between two cars, so you're counting the miles driven by both cars, but only counting the collision between them as 1. So you'd be off by a factor of 2 if you compare that to a distance driver by a single car divided by its collisions.

Yes, thank you, definitely correct. So it's more like 4 accident/million miles from this dataset, although another poster made a good case that Waymo's data includes minor accidents that aren't in the data I looked at, which would make 12 accidents/million miles a better benchmark.

But is it 2.08 crashes/million miles of driving around extremely easy and empty suburbs? There's a reason they started where they have - it's the easiest driving in the world. That's very sensible but it does mean you can't just compare statistics like that.

They are also counting incidents where the relative speed was 1mph, or a person walked into the car. Perhaps fatalities per billion miles would be a more comparable number.

* Seems like they're far from human performance.

* On highways while under the supervision of a trained driver and autonomously within a select area.

You also need to account for confounding variables, such as geography. Does Phoenix mirror the nation as a whole?

Did you count all incidents or only the ones that happened in real life and not the simulations?

I omitted the simulations in the Waymo count. Good idea to add them in?

Yeah you should include them because they were only prevented by a safety driver who won't exist in the long term.

Thanks! I missed that definition in the article. Updated the ratios. So it's actually 3.7x more accidents not 1.5x. Definitely changed my conclusion.

Simulations counts were times when a safety driver took control of the vechile, and post-incident simulations with the same data show that the AI would have crashed the car without the safety driver. So fair to include them.

Reply on news.ycombinator.com