Tens of thousands of Americas are killed every year in car crashes. The existence of a case or some cases where a self-driving car caused injury has zero value. What matters is the rate of cases per mile driven.
The same of course applies to self-driving cars; they are literally cars driven by software, of course you need to do a root cause investigation every time to rule out that it's not a bug in the software that will kill another person (and many after) when the next car happens to go down that rare branch of the system. It's embarrassing to see that the people that call themselves engineers at these companies have not held their work to this standard, and are instead publishing glossy brochures making whacky statistical arguments.
I've personally read through the root cause reports for most of the notable AV accidents. They're not always quite as intensive as aerospace, but I'd be hard pressed to describe any of them as wacky statistical arguments.
Obviously most of those reports aren't public, but I'm assuming you also have industry access.
"The number of pedestrian deaths in the United States is skyrocketing. In 2022 traffic crashes killed 7,805 people on foot—that’s an 83 percent rise from 2009, and a 40-year high. The vast majority of those deaths involved a car colliding into a human"[1]
"Pedestrian deaths have been climbing since 2010 because of unsafe infrastructure and the prevalence of SUVs, which tend to be more deadly for pedestrians than smaller cars, according to Martin."[2]
The issue with the safety claims of self-driving is focusing on vehicle-to-vehicle interactions and failing to handle the chaotic and unpredictable nature of the environment. There are numerous stories of pathological behavior of self-driving vehicles when encountering simple environmental features that a human driver would handle without a second's hesitation[3]. Pedestrians, cyclists, people using mobility devices, and numerous other non-vehicle road users represent an unaddressed challenge to the safety claims made by Waymo and others.
1 https://slate.com/business/2024/10/cars-suvs-pedestrian-deat...
2 https://www.npr.org/2023/06/26/1184034017/us-pedestrian-deat...
3 https://www.npr.org/2023/08/26/1195695051/driverless-cars-sa...
CDC “Underlying Cause of Death” dataset sez https://wonder.cdc.gov/ucd-icd10-expanded.html https://i.imgur.com/4PB0xyC.jpeg
- “Person injured in unspecified motor-vehicle accident, traffic” is the 50th leading cause of death at 0.4% of deaths.
- “Person injured in collision between other specified motor vehicles (traffic)” is the 108th leading cause of death at 0.2% of deaths.
0.4% and 0.2% sound low, but make up for ~110,000 deaths. Spread across a 5 year period does indeed equal “tens of thousands” every year.
But, more importantly, you missed a bunch of relevant categories:
V89.2 (Person injured in unspecified motor-vehicle accident, traffic) 80,434
V87.7 (Person injured in collision between other specified motor vehicles (traffic)) 29,982
V09.2 (Pedestrian injured in traffic accident involving other and unspecified motor vehicles) 27,934
V03.1 (Pedestrian injured in collision with car, pick-up truck or van, traffic accident) 15,129
V43.5 (Car occupant injured in collision with car, pick-up truck or van, driver injured in traffic accident) 9,810
V29.9 (Motorcycle rider [any] injured in unspecified traffic accident) 8,410
V29.4 (Driver injured in collision with other and unspecified motor vehicles in traffic accident) 7,688
V47.5 (Car occupant injured in collision with fixed or stationary object, driver injured in traffic accident) 6,379
V49.9 (Car occupant [any] injured in unspecified traffic accident) 6,349
V23.4 (Motorcycle rider injured in collision with car, pick-up truck or van, driver injured in traffic accident) 5,851
V43.6 (Car occupant injured in collision with car, pick-up truck or van, passenger injured in traffic accident) 3,728
V27.4 (Motorcycle rider injured in collision with fixed or stationary object, driver injured in traffic accident) 3,504
You do not get to counter-argue: “What matters is the rate of cases per mile driven” without actually presenting that number with supporting evidence. Otherwise the only sound conclusion is the default presumption of non-safety.
In the case of Waymo, we have some tentative supporting evidence from this and other studies Waymo has run. However, that is still insufficient, even ignoring the lack of audits by non-conflicted parties, to strongly conclude Waymo is safer than a human. The evidence is promising, but it is only prudent to wait for further confirmation.
In contrast, Cruise was almost definitely not safer than a human driver.
In 2023, Cruise ADS cars drove 2,064,728 miles [1] and were involved in, by my count, 29 collisions with 5 causing injury [2], namely incidents on 2023-05-04, 2023-05-21, 2023-06-09, 2023-08-18, 2023-10-02.
That is ~72,000 miles per collision and ~400,000 miles per injury in contrast to the national human averages of ~500,000 per reported collision (which is non-comparable) and ~1,270,000 miles per injury (which is comparable). So, absent a more detailed analysis, Cruise ADS cars were ~3x MORE likely to be involved in a injury causing collision per mile.
Details and evidence matter in these discussions. Blanket rhetoric and optimism is not prudent when discussing new safety-critical systems.
[1] https://thelastdriverlicenseholder.com/2024/02/03/2023-disen...
[2] https://www.dmv.ca.gov/portal/vehicle-industry-services/auto...
> You do not get to counter-argue: “What matters is the rate of cases per mile driven” without actually presenting that number with supporting evidence. Otherwise the only sound conclusion is the default presumption of non-safety.
I then pointed out how Waymo does present such evidence. But, if you applied that argument to Cruise you would be wrong. That demonstrates how that argument (when not presenting the numbers) can be used to support both good and bad and is thus a bad argument.
The correct argument when somebody points to anecdotes of bad outcomes is to present statistically sound data of good outcomes, not argue they did not present statistically sound data of bad outcomes thus you get to assume it is good.
Waymo releases its safety data: https://waymo.com/safety/impact/, which is backed by public reporting requirements.
To say that it is wholly insufficient to make any safety claims on publicly driven 50M miles, is ridiculous. At the very least, it appears sound, robust and transparent, and able to be validated.
> https://waymo.com/blog/2024/12/new-swiss-re-study-waymo
Is Swiss Re a valid third party? They also address peer-reviewed and external validation in the above safety impact page.
I can understand being skeptical because of Cruise and especially claims made by Telsa, but there is a preponderance of supporting data for Waymo.
Given all of this evidence, you would still conclude Waymo is unsafe?
> In the case of Waymo, we have some tentative supporting evidence from this and other studies Waymo has run. However, that is still insufficient, even ignoring the lack of audits by non-conflicted parties, to strongly conclude Waymo is safer than a human. The evidence is promising, but it is only prudent to wait for further confirmation.
You are not making a distinction between concluding unsafe and not being able to conclude safe. It is standard practice to not presume safety and that positive evidence of safety must be presented. Failure to demonstrate statistically sound evidence of danger is not proof of safety. Failure to disprove X is not proof of X. This is a very important point to avoid fallacious conclusions on these matters.
To discuss your specific points. Yes, the data is promising, but it is insufficient.
Traffic fatalities occur on the order of 1 per 60-80 million miles. Waymo has yet to reach even one expected traffic fatality yet. They appear to be on track to doing better, but there is not enough data yet.
The reports Waymo present are authored by Waymo. Even the Swiss Re study is in cooperation with Swiss Re, not a independent study by Swiss Re. The studies are fairly transparent, they point to various public datasets, there are fairly extensive public reporting requirements, and Waymo has not demonstrated clear malfeasance, so we can tentatively assume they are “honest”. But we have plenty of examples of bad actors such as Cruise, cigarette companies, VW , etc. who have done end-runs around these types of basic safeguards.
Waymo operational domain is not equivalent to standard human operational domain. They attempt to account for this in their studies, but it is a fairly complex topic with poor public datasets (which is why they cooperated with Swiss Re) so the correctness of their analysis has not been borne out yet. When Waymo incorporates freeways into their public offerings this will enable a less complicated analysis which would lend greater confidence to their conclusions.
Waymo is still in “testing”. As their processes appear to be good, we should assume that their testing procedures are safer than should be expected out of actual deployment or verification procedures. That is not a negative statement. In fact, it would be problematic if their “testing” procedures were less or even equal in safety to their deployment procedures. That is just how testing is. You can and must apply more scrutiny to incomplete systems in use and prevent increased risks especially while under scrutiny otherwise you are almost certainly going to be worse off in deployment where there is less scrutiny. We have yet to see how this will translate out to deployment, so we will need to wait and see if safety while under test will appropriately apply to safety while in release. This is analogous to improved outcomes for patients in medical studies even if they are given the placebo because they just get more care in general while in the study.
Anyways, Waymo appears to be doing as well and honestly as can be determined by a third party observer. I am optimistic about their data and outcomes, but it is only prudent to avoid over-optimism in safety-critical systems and not accept lazy evidence or arguments. High standards are the only way to safety.
Counter-point: "That's false because Cruise had an accident for which they were at fault".
OP: "The existence of a case or some cases where a self-driving car caused injury has zero value. What matters is the rate of cases per mile driven."
You: "You do not get to counter-argue."
Yes, they do. OP's point is valid. One can't refute the original assertion by citing one accident by another company. It's a logical fallacy (statistically speaking), and a straw-man (Waymo can't be safe, because other self-driving cars have been found at fault). The validity of the original claim has nothing to do with an invalid counter-claim.
> However, that is still insufficient, even ignoring the lack of audits by non-conflicted parties, to strongly conclude Waymo is safer than a human.
When you have a large, open, peer-reviewed body of evidence, then yes, that's exactly what you get to claim. To reject those claims because Waymo was involved is ad-hominem. It's not how science works. It's not how safety regulations or government oversight works. If you think it's insufficient, you can attack their body of work, but you don't get to reject the claim because they haven't met some unspecific and imaginary burden of proof.
50% of fatalities involve alcohol or drugs and are often single vehicle accidents.
25% involve youth or inexperience.
15% involve motorcycles.
15% involve pedestrians.
What I really need to see is a complete breakdown of every accident a Waymo has had. Then I can start to compare their actual performance to the previously known outcomes.
No, that's not how statistics works.
The percentage data's accuracy depends mainly on the number of incidents recorded (and somewhat on the rate of incidents). But the percentage of the whole is completely irrelevant.
If you are basing something on 10 incidents but it's 50% of the total, it's still terrible accuracy.
Whereas if you are basing something on 100,000 incidents but it's only 0.1% of the total, it's still going to be quite accurate, assuming the incidents come from the same overall distribution.
If the user base of "waymo riders" and "everyday drivers" does not match then you're not sampling what you think you are.
size along is not enough. The sample may be biased and it certainly is in Waymo case e.g., roads/vehicle conditions.
The ratio of 3.2 trillion to 56.7 million, which is already incredibly generous to Waymo's position, is 5 orders of magnitude in difference. So any calculations from Waymos data are going to be insanely inaccurate and not something you can extrapolate from.
The main, and most obvious case, evidenced by this, is Waymo does not operate where snow falls. Human beings do.
We're missing so much of the picture I don't think you can say Waymo's are 75% less accident prone, or 80% less likely to hit a pedestrian. Those are just nonsense numbers.
You could still say you care about snow driving and want to see that comparison, but it doesn't mean the claims in this paper are wrong.
Your third sentence doesn’t follow from your first two. On what grounds do you draw this conclusion?
This suggests that Waymo is cutting traffic fatalities by 50% (per million miles) right off the top.
Drunk people being known for having exceptionally poor judgement and self awareness.
It suggests that they _could_ cut fatalities by that much. Then again, a whole new mode of accident, where the inebriated decide to step out of a moving vehicle and injure themselves that way.
This is a dynamic system where human decisions are never fully removed from the loop.
If I understand correctly, you believe that the advent of self-driving cars will cause passengers to voluntarily exit a moving vehicle? That sounds like absolute nonsense with no basis in reality.
Why would they need to be in a self-driving vehicle to do that?
> Building on that, the Collision Avoidance Benchmarking paper presents a novel methodology to evaluate how well autonomous driving systems avoid crashes. The study, which to our knowledge is the first of its kind, introduces a reference model that represents an ideal human state for driving—the response time and evasive action of a human driver that is non-impaired, with eyes always on the conflict (NIEON). Put simply, unlike an average human driver, NIEON is always attentive and doesn’t get distracted or fatigued¹. The data showed that the Waymo Driver outperformed the NIEON human driver model by avoiding more collisions and mitigating serious injury risk in simulated fatal crash scenarios.
(From https://waymo.com/blog/2022/09/benchmarking-av-safety)
AIUI (I'm not on that team), a major challenge is getting good baseline data. Collision reports may not (reliably) capture that kind of data, and it's clearly subjective or often self-reported outside of cases like DUI charges.