Hacker News Reader

Comment by crazygringo | Hacker News Reader

> The US drives about 3.2 trillion miles per year. Waymo has 56.7 million miles over several years. Their percentage data is essentially useless.

No, that's not how statistics works.

The percentage data's accuracy depends mainly on the number of incidents recorded (and somewhat on the rate of incidents). But the percentage of the whole is completely irrelevant.

If you are basing something on 10 incidents but it's 50% of the total, it's still terrible accuracy.

Whereas if you are basing something on 100,000 incidents but it's only 0.1% of the total, it's still going to be quite accurate, assuming the incidents come from the same overall distribution.

View on HN

This! (Thank you for the comment). There’s a reason a 1000 random samples is adequate to reasonably estimate what’s common metrics in a population the size of USA or India (or infinitely large).

Random samples of the _same_ user base.

If the user base of "waymo riders" and "everyday drivers" does not match then you're not sampling what you think you are.

Yeah, that’s fair, but is implicit since I’m arguing against the “sample size is inadequate” POV, not the “there are distributional biases in data” POV. There are a gazillion ways to adjust for these biases (ex. propensity score matching) going beyond just user-base but also including weather type, road type, location, time of day, day of week, traffic density, pedestrian density … that can be done easily with far less than the sample size waymo has. And I bet they do these adjustments.

It reminds me of a large sample in the Reader’s digest poll: https://en.wikipedia.org/wiki/The_Literary_Digest#Presidenti...

size along is not enough. The sample may be biased and it certainly is in Waymo case e.g., roads/vehicle conditions.

> If you are basing something on 10 incidents but it's 50% of the total, it's still terrible accuracy.

The ratio of 3.2 trillion to 56.7 million, which is already incredibly generous to Waymo's position, is 5 orders of magnitude in difference. So any calculations from Waymos data are going to be insanely inaccurate and not something you can extrapolate from.

The main, and most obvious case, evidenced by this, is Waymo does not operate where snow falls. Human beings do.

We're missing so much of the picture I don't think you can say Waymo's are 75% less accident prone, or 80% less likely to hit a pedestrian. Those are just nonsense numbers.

The paper under discussion only considers human accidents in similar environments to where Waymo operates. So it's only making a claim about like-for-like driving.

You could still say you care about snow driving and want to see that comparison, but it doesn't mean the claims in this paper are wrong.

Reply on news.ycombinator.com