Hacker News Reader

Comment by paxys | Hacker News Reader

I have felt since day 1 that "autopilot is better than the average driver!" is the most misleading fact/statistic in the industry.

The "average driver" involved in an accident includes drunk drivers, road ragers, habitual speeders, teenagers, people driving in bad conditions (rain, snow), people driving without sleep and several other risky groups that most people buying Teslas aren't part of. To be worth it, the car has to reduce my personal accident risk under the conditions I usually drive, otherwise I prefer to keep my own hands behind the wheel over handing over control to an algorithm.

View on HN

I don't disagree with your broader statement about autopilot's safety relative to other drivers, but...

>The "average driver" involved in an accident includes drunk drivers, road ragers, habitual speeders, teenagers, people driving in bad conditions (rain, snow), people driving without sleep and several other risky groups that most people buying Teslas aren't part of.

... what? Tesla owners absolutely can be habitual speeders, people who drive without sleep, road ragers, and drunk drivers. Can you please explain your thought process behind this claim? I really don't understand why anyone would assume that Tesla owners are less likely to have those traits than non-Tesla owners.

Yeah, seems like a typo a GPT-3 would do.

The claim seems to be that Teslas do not drink before driving, not Tesla owners, right?

Because they are buying an expensive, new car, meaning higher income, meaning that those things are less likely : https://academic.oup.com/alcalc/article/46/6/721/129644

In my person experience, the "drunk drivers", "road ragers", "habitual speeders" groups seems to be over-represented in expensive cars rather than cheaper cars. Poor people cannot afford to crash their car while rich people can.

Data released by the DOL and insurance companies does not match with your personal experience. There's a reason 18 year olds driving shitty cars pay through their nose for liability coverage.

I do not think a car crash is normally the result of a decision people make based on whether they can afford it or not.

Keep in mind that this study could just mean that people in expensive cars with higher incomes are less likely to be arrested/stopped by police.

Hmm.. The underlying studies (Baum, 2000; Eensoo et al., 2005) that found links between socioeconomic status and DUI/DWI did so by comparing those arrested against a control group – which I believe should deal with that concern?

For your specific question, actuarial tables for car accidents/deaths are available publicly, and the profile for the typical Tesla owner (middle aged, high income, driving expensive sedan/SUV) is considered much safer than average.

Moreover it is true in general that most people driving aren't drunk or high, aren't reckless etc., and so most Tesla drivers aren't either. The number of accidents isn't uniformly distributed among the public.

More broadly though, I never said that zero Tesla owners (or any other specific car drivers) do any of these things, but that using autopilot with an above average safety record can still mean that the personal safety of an individual or group of individuals goes down. For the system to be a net benefit it has to be pushed to the below average drivers.

I read it that way at first too but I think the reasoning is actually correct. The population of Tesla drivers will of course resemble the overall population of drivers but most drivers are not sleepy/raging/drunks and by extension “most people buying Teslas aren't part of” that group either.

To use an analogy: the average person has 1.9 legs but a product which results in its user having 1.95 legs is not an improvement for most people.

Wow. Excellent analogy.

Great analogy, thank you, ive been struggling to put this across

For an individual person to be willing to adopt autopilot on safety grounds, autopilot has to be safer than that individual person's driving, not just safer than the average person's driving.

If your driving skill is above the mean (because you're never a road rager, speeder, etc.), then you're worse off using Autopilot, even if Autopilot is as safe as the average driver.

Since most people probably consider themselves above average drivers (and in fact most people may be above the mean if the bad drivers are outliers), this limits the number of people who will believe that Autopilot makes them safer.

> Tesla owners absolutely can be habitual speeders, people who drive without sleep, road ragers, and drunk drivers.

The average Tesla owner may well be those things. But I know that *I* am not those things.

I know my own driving record well. I know my own accident history well.

So you don't have to convince me that Tesla is better than the average driver. You don't have to convince me that Tesla is better than the average Tesla owner. You have to convince me that Tesla is better than *me*.

Now, apply that same analysis to the 88% of drivers that think they are above the median [1] in terms of driving safety. It doesn't matter if I am _right_, it matters what I think about my relative driving safety.

[1] https://www.gwern.net/docs/psychology/1981-svenson.pdf

>I really don't understand why anyone would assume that Tesla owners are less likely to have those traits than non-Tesla owners.

I think it was a comparison to Autopilot rather than Tesla owners. Robot drives can't be drunk, don't need sleep etc.

If I remember correctly, they also cover the easy part of the road and make the human take over at tricky situations.

So at the end of the day, you have spotless records for the autopilot that was driving many many miles on the straight line and humans having accidents at short intersections or construction sites. This translates into very favorable numbers for autopilot when presented as accidents per miles driven.

If it were the case that autopilot was benefiting from dumping all of the difficult parts on the humans, you would expect to see much higher than average rates of accidents per mile on the human driven parts, because they're only getting the hard parts. This tends not to be true, so I'm doubting it's a real effect.

Humans don’t disengage and let the machines handle the situation but machines do it all the time. Do the machines disengage at tricky situations or straight lines?

The statistics are there to read however, and suggest your reasoning is not correct.

Case A: Non Tesla Human drives 999 easy miles and 1 hard mile

Case B: Autopilot drives 999 easy miles, and human drives 1 hard mile

If the effect size of the hard mile is so large that its skewing the statistics, you would expect the Telsa human driver to have a horrendous per mile accident rate relative to the non tesla human driver. What's most likely is that the accidents following a human handoff get correctly allocated to the Autopilot and the effect size being described is not actually that significant.

stats for easy miles and hard miles? link please?

Humans don’t crash at every tricky situation but Tesla claims that humans are horrible drivers and their cars gives the control to the humans when when something happens.

Literally any distribution of hard miles and easy miles will produce the same outcome in this thought experiment, to varying effects, if your premise of Tesla hiding the AI's incompetence with passing off to humans when driving is challenging is accurate.

If your premise is to be believed as a significant effect, you must also accept that this outcome should be visible in the data

Student drivers (with an instructor in the car, and secondary controls) have very few accidents, if any, not because of their safety record but because every mistake is saved by the instructor. A car can be considered as driving "safer than a human" when it can match the average human in any conditions the average human drives. Everything else is just squinting at the data and choosing an interpretation that fits your personal opinion.

A Tesla can relatively safely cover traffic like divided or controlled access highways. That's a very narrow slice of all possible driving situations and not one responsible for most accidents.

So as a thought experiment, if Autopilot + Human intervention reduced the rate of accidents by 50% vs. just humans after normalization; can we consider autopilot to be adding value?

Of course the Autopilot adds value, all such driver assists are there to add value and help the driver on any car. I just don't think Autopilot can drive, let alone drive safer than a human. There's a difference between "helping a human drive safer" and "driving safer than a human". This is a confusion many people make when reading these stats, to Tesla's benefit.

Also the oldest Tesla with AP is less than 10 years old (with an average of 3-4 years given the sales trends). The average age of cars on the street in the US is over 12 years old. An accident statistic that looks at reasonably new, premium cars against everything else won't paint the correct picture.

If that means the supervision actually works, then it doesn't sound like there's a problem?

> Student drivers (with an instructor in the car, and secondary controls) have very few accidents [...] because every mistake is saved by the instructor

Is that the only reason? I suspect that many (most) student drivers are also in general more alert, drive slower, and follow each rule/guideline to an extreme. If they don’t the instructor will probably end the lesson and remove a dangerous driver of the road.

All student drivers particularly at the beginning are "dangerous drivers" because they aren't drivers but they're asked to drive. In their first half of driving school most students would probably cause an accident within minutes if not for the instructor. I live on a quiet street where most driving schools take their students due to the almost non-existing traffic. I can't count how many times I see the instructor having to intervene to avoid a crash even at simple things like keeping a straight line or a right turn (too tight or too wide).

Is this data available anywhere? How do I know that non-autopilot Tesla mile per accident rates aren't horrible?

Tesla publishes their own data for the safety of autopilot, which I presume is based on their own analysis of accident records. Is this same detailed information available to other groups (insurers, NIST, etc)? Or do they just calculate an aggregate "Tesla mile per accident" rate that is a blend of the great autopilot rate and horrible human rate?

I'm not trying to be facetious here. I have no idea if this data is available to any groups other than Tesla themselves. And if so, do they publish those numbers?

Tesla releases the miles per crash rates quarterly, for autopilot and non autopilot cases. Autopilot crashes include anything within 5 seconds of disengagement. The human rate tends to be more than 2x worse than the autopilot rate. This is not normalized for factors like road context.

The human rate for tesla driven miles tends to be ~4x better than the other brands' average. To precisely answer this question you would want to see both a comparable brand's humans' performance; and probably the split of humans who used autopilot and humans who don't ever use autopilot. We don't have that, but in my personal opinion there's enough evidence to suggest it's probably not a grand conspiracy. I'm of the opinion that autopilot being ballpark on par with other drivers is more than enough to reduce accidents substantially, at scale.

https://www.tesla.com/VehicleSafetyReport

It’s not just that. There’s a context switch when the autodrive disengages, so the human is actually less ready for the hard mile than if they were driving the whole time. Sure m, the human is supposed to be able to take control at anytime, but I don’t think that really happens. The whole purpose of autodrive is do you don’t have to pay attention and drive.

Could you provide a source? That sounds fairly interesting.

Just some reasoning really. Statistics and proper normalization are hard.

Tesla tends to say that autopilot crashes occur 1/2 as often as non autopilot crashes. That's likely not normalized to road conditions. But if you assume that Tesla is secretly just putting all the hard miles on the humans, then that would imply humans are driving many more hard miles and should have higher accident rates. The autopilots meanwhile must be performing worse on the easy miles and racking up additional accidents that wouldn't have otherwise happened.

If you combine those two, the overall rate of accidents should be higher than average, but it's actually lower by a fair margin. Again, normalization is hard.

Ideally you would be able to compare human drivers of another comparable car brand to the human drivers of Tesla to confirm the Tesla drivers don't seem to be being judged on unreasonably difficult conditions.

There was a sourse but I could not find it ATM. It’s fairly simple, people don’t disengage and their driving safety is judged over all the miles they drive + all the situations where Autopilot disengages.

Tesla Autopilot is judged only by the miles driven without disengagement, which is quite limited actually. You can watch Youtube videos to see at what kind of situations Tesla autopilot gives up.

There’s no situation where the Autopilot takes over from the human saying “That’s a tricky road, let me handle it”.

You seem to be missing the point though. If this were significant, then human tesla drivers should be shown as performing much worse than other car drivers, because you're claiming they have a disproportionately large riding time in "tricky roads".

A non tesla driver should be doing way better because they get to pad their score with the easy roads the autopilot supposedly gets.

Maybe that’s the case, Tesla data isn’t public. They don’t publish data but conclusions and their conclusions are questionable.

Actually its more an issue that other car companies don't publish their data for comparison.

On a dumb average tesla is way better, but it'd be more compelling if we could compare to new luxury brands with similar target markets

How it's other car companies fault that Tesla isn't publishing data so we can check if Autopilot miles are mostly on straight lines and human miles are at tricky situations?

Tesla publishes the human accident rate, and the autopilot accident rate.

If another comparable car company published their human accident rate, and you were willing to assume that the baseline human accident rate should be the same for both brands, you could evaluate your hypothesis that Tesla is shoving the tricky situations onto humans.

As mentioned elsewhere, just because a situation is difficult for FSD to parse and process doesn't inherently make it a dangerous situation for a human driver.

Your conclusion relies on the - quite faulty - assumption that "situations that inherently difficult for FSD to handle" are automatically "also more dangerous for human drivers". In snowy conditions, humans do just fine, generally, at following "lanes", be they the actual lane, or the safest route that everyone else is following. Humans are also capable of deducting lane direction, orientation, even when there are contradictory/old lane markings on the road, a situation FSD regularly causes danger in.

Or that that negative effect is lost in the orders of magnitude of "all human drivers across all miles" versus FSD.

You don't think that human driver accident rates in snowy conditions are much higher than in fair weather conditions?

I was referring to the lane "following" aspect. FSD disengages, but most humans can follow lanes in snow. Of course there are separate concerns.

I've been using taxis in my city for like 50 times, and I've never even felt remotely in danger, except when I was in a Tesla taxi. The driver were well composed, calm and seemed rational, but we've had 5 (!) near crash situations in a 15 minute drive, simply because the car can accelerate into spots opening up in moderately dense traffic so quickly that nobody can expect it.

The car enables a completely novel driving style which people need getting used to.

> several other risky groups that most people buying Teslas aren't part of.

This is a fallacy. Ask any demographic you like if they're road ragers, drunk drivers, etc... and they'll deny it. Everyone is sure that rare and terrible outcomes will never happen to them because of their own behavior, including the people to whom it happens!

It's probably true that middle aged premium car owners are safer on average, but you don't get to rule that stuff out by fiat. In fact there have been a few "Autopilot Saves Passed Out Driver" stories over the past few months, where Tesla drivers were clearly impaired.

As for this particular paper: this is just P-hacking folks. You can't take a vague dataset[1] and then "correct" it like this without absolutely huge error bars. Why correct based only these variables? Why did they all push the results in one direction? Why not gender? Why not income? Why not compare vs. like cars?

This isn't good statistics, it's just more statistics. If we want the real answers we should get a better data set (which, I'll agree, would likely involve some regulatory pressure on manufacturers).

[1] And, to be fair: Tesla's safety report is hardly comprehensive and provides no data other than the aggregate numbers.

I just think a lot of Tesla drivers see themselves as elite drivers or something when we all know that's not true if you step back away from the situation.

> You can't take a vague dataset and then "correct" it like this without absolutely huge error bars.

That ship has already sailed; Tesla makes active safety claims based on that dataset. To hold research to your standard here would be to say that Tesla can make its claims, but nobody can challenge those claims.

> Why did they all push the results in one direction?

If you have two correction steps, then a priori you'd expect a 25% chance that they both act in the same direction. I don't think this is very remarkable.

> Why not gender?

This probably would be a reasonable addition. I doubt it would change the results much, but we have the well-known fact that insurance rates differ for men and women, so it may be relevant.

> Why not income?

First, is this data even generally available? If the data doesn't exist, then we can't control for it.

Second, should we expect crash rates per mile driven to differ greatly by owner/driver income? A priori, I wouldn't think this demographic quality to have a strong impact.

> Why not compare vs. like cars?

I think "personal vehicle" is a reasonable comparative category. Would we expect collision rates to differ greatly between more specific categorizations? For the sake of the overall conclusion, would we expect Tesla-equivalents to be particularly crash prone in the broad dataset?

Any statistical analysis will have its limitations, but when we're talking about life-safety claims from a manufacturer I think we should have wide latitude to look at critical evaluations of the original data.

> That ship has already sailed; Tesla makes active safety claims based on that dataset. To hold research to your standard here would be to say that Tesla can make its claims, but nobody can challenge those claims.

I don't follow that logic. Claiming that Tesla's data set is incomplete is fine.

That doesn't mean that any junk science that refutes it must be correct. Bad statistics is bad statistics. This kind of "correlate dissimilar data sets by using more dissimilar data" is a terrible way to do science. Quite frankly it's almost guaranteed to be wrong. There are regulatory bodies out there with access to real data. Get them to figure it out.

In my case, I'm happy to engage Autopilot even though I'm under no illusion that it makes me safer. It's just a very useful convenience feature.

OTOH: Tesla could still claim that if you replace all the distracted/drunk drivers in beat-up cars with new Teslas running Autopilot, the roads would be safer.

But that's not really contradictory, is it? If you believe that you're a better driver than autopilot, just don't use it, or buy another car. It's not at all dishonest or deceptive marketing (if the claim is factually accurate, of course).

I agree with this, and have also been accused of being a Tesla hater, but OTOH someone has to pave the way for the future in which none of /those/ people have to drive either, and our cities aren't taken over by surface parking for cars that move two or three times a day at most.

If we're not gonna switch to sane urban planning, may as well bring on the robo-taxis that can go anywhere

Confounding factors matter.

E.g. I would not be at all surprised if drivers of high-end BMWs have lower deaths per mile due to older population, more affluent, etc.

So how much better than the median randomly assigned uber driver does it need to be?

So basically everyone in a car? :)

I think it's fair to compare to the average driver. You may be not average by virtue of not being a teenager and not driving drunk, but you're not magically immune to being run over or crashed into by one of those. So even if a hypothetical FSD capable car would not drive one yota better than you do, it cwould still make you safer by virtue of making the surrounding environment safer.

Sure the worst drivers out there utilizing assist tech like autopilot would benefit everyone, but data shows that they aren't the ones spending money on fancy cars. My comment was about me making a decision for myself.

Plus in your hypothetical the autopilot can drive at least as well as me, which is also a big assumption and not something I believe with what I have seen so far.

It can't be just any "average driver", they should be compared against "average driver" who just bought $40-50K NEW car.

>> "average driver" who just bought $40-50K NEW car.

Driver and buyer/owner are different things. Few teenagers ever buy a Tesla, but they certainly drive them. A car cannot only be safe in the hands of the rich initial buyer who lives in a nice climate. It must be safe in all the other people who that owner may let drive. It must also be safe for subsequent second and third owners, people who might not be wealthy enough to stay home when it rains or when the autopilot thinks conditions are too rough. (It was -35c on my drive to work this morning. Dark. Ice fog then blowing snow.)

Uhm, I don't know what insanely rich middle eastern country you're from but in most of the world teenagers do not generally drive Teslas!!

Have you been to LA, SF, Vancouver or Denver? These are Teslas, not Ferraris. Lots of teenagers are driving their parent's tesla. Go look at any university parking lot and you will find plenty of 50,000$ cars.

Exactly right. The population has to be compared to people driving new Volvo S90 or M-B C300, cars that have IIHS fatality rates of zero.

If you are speaking about the safety features in a new car, then I won't disagree. However, if you are implying that those with a nice new car are somehow better drivers, more attentive drivers, or care more about their car then I would highly disagree. As someone who purchased a new car last year, and does greatly care about it, I'm routinely shocked by people and their clear disregard for their own vehicle. Reckless driving, texting while not even attempting to look, and the way people park their cars so close to you that they can't even get out* have all shocked me now that I'm paying more attention.

* I have even stared at this guy struggling to get out of his fancy Jeep who opened his door into my car. I know this anecdotal, but all it takes is a few counter examples to show that car price etc does not have a super high correlation to how careful & considerate you are.

Look at the accident and fatality rates for drivers of luxury cars. They are much, much lower than the same rates for less expensive cars. In fact, if you break it down by vehicle model, there are several models with 0 fatalities at all.

I'd be curious to see where you are able to find good data for that. To be honest, I haven't found data that shows a clear difference between luxury and non-luxury cars

i.e. https://quotewizard.com/news/posts/most-accident-prone-cars lists some luxury brands as having high rates - "Luxury sport car brands Infiniti, Lexus and Acura are among most accident-prone cars on the road."

and if you look through this data from IIHS https://www.iihs.org/ratings/driver-death-rates-by-make-and-... there are plenty of luxury cars with high death rates.

Tesla's claim relied on inclusion of motorcycles one time, which seems super misleading and unfair.

Misleading, from a company that once (several days ago) claimed that they sold more cars in Australia than Camry's last year.

Until someone pointed out that vehicle registration... disagreed. By nearly 30%.

Then "Oops. We made a mistake and counted 3,000+ deliveries which haven't actually been, well, delivered."

I hope they get spanked hard for that. Australia has very onerous "truth in advertising" laws.

Given what I’ve been hearing/reading about autopilot I’m more worried about being hit by one of them than a human driver.

No.

I'm not going to boast about how good of a driver I am (I am!!), but lets say I am a median driver. I would want a FSD to be better than a median driver. Not the average crash/mile stats that include all kinds of long tail idiots doing the most damage.

    * ... alcohol-impaired driving crashes, accounting for 28% of all traffic-related deaths in the United States.
    
    * Drugs other than alcohol (legal and illegal) are involved in about 16% of motor vehicle crashes.

https://www.cdc.gov/transportationsafety/impaired_driving/im...

So 44% intoxicated to some degree (unless the stats overlap). That number is not even including reckless drivers not driving under influence, teenagers that just got the license, bad health elderly etc.

My problem is that the median driver is much much better than the mean driver and tesla os comparing against means.

The damage done in a car distribution is heavily skewed towards people that get in fatal crashes or total a car. I'd randomly guess like 20-30% of drivers are below mean. Most people won't total a car, vast majority will never kill anyone.

If you get median people into an autopilot car that's got a mean safety record we end up pulling the average down, the road becomes less safe. And luxury sedan buyers tend to be one of the safest demos on the road, which makes the problem even worse.

Median vs mean of what measure?

Huh. Citation needed that drivers who buy, say, a Model S, particularly one with Ludicrous mode are less likely to be "habitual speeders" than the average driver.

This has come up before. When the new FSD beta started, people started claiming that safe driving was a function of vehicle price, and therefore Tesla drivers, especially those who had paid for FSD, were more likely to be safer. When I noted that my current vehicle costs more than a Model S, based on the logic, Tesla should be recruiting me to beta test FSD, well it was hard to find a refutation.

Reply on news.ycombinator.com