Hacker News Reader

Comment by pja | Hacker News Reader

And no, you can't get out of culpability by claiming statistical superiority.

Quite. Tesla is deliberately comparing their modern vehicle design & wealthy driver base (statistically one of the safest cohorts) with the entirety of the U.S. driving population. This is bad statistics: We expect Tesla vehicles to have far fewer accidents than the mean US vehicle per mile, because Tesla's are expensive modern vehicles with (relatively) wealthy drivers who can afford to keep them maintained & will themselves be in better health than the mean US driver.

Don't tell us how safe Tesla's are compared to the entire US driving population: Tell us how safe they are compared to equivalent vehicles from other manufacturers. My working assumption is that Tesla doesn't do that because it would be far less flattering to the 'A Tesla is totally safe, honest' PR boosterism that runs through every Tesla press release.

View on HN

> Don't tell us how safe Tesla's are compared to the entire US driving population: Tell us how safe they are compared to equivalent vehicles from other manufacturers.

I'm having a hard time to find an exact number comparison, but:

- Tesla claims 1 fatality per 320M miles

Compared to

- A 2015 study of 2011 model year vehicles showed, for instance, Volvo XC90 had never been involved in a driver fatality (1)

- There are about 10K 2011 XC90s in the US (2)

- Avg. US drivers cover 13,400miles / year (3)

So:

- 4 years studied * 10K vehicles * 13,400 avg miles -> 536,000,000 fatality-free miles

This is obviously super hand-wavy, but I think it's fair to state as a hypothesis that Tesla could be twice as deadly as the safest high-end vehicles.. you'd need to design some experiment to attempt to falsify it to feel confident this is correct.

(1) https://www.freep.com/story/news/nation/2015/01/29/study-cha...

(2) http://carsalesbase.com/us-car-sales-data/volvo/volvo-xc90/

(3) https://www.fool.com/investing/general/2015/01/25/the-averag...

Probably should also note, that the average Tesla driver is likely male, has money (i.e. more risk taking), etc. Essentially, compare to sports cars as opposed to high end vehicles.

They we're originally advertised as sports cars really. 0 to 60 faster than any other car, etc.

Eh, you can't cherry pick specific (model, year) combinations after the fact without overfitting. You expect some variation in deaths per (model, year) and by searching for the least deadly ones, you'll dramatically influence your estimate simply because you used a maximizing search instead of a broad survey of a-priori safe models.

This is totally true, and I agree; I tried to guard against this at the end with the "I think this is enough to state a hypothesis, which would then need to be tested.." paragraph.

However - the same is true for 2009, 2010, 2011, 2012 model XC90s, and they are just one of 10 models that had no fatalities in those four model years in the study. I just picked that year and model to get one data point for mileage to compare; the other model years of XC90 have similar sales, so the same applies.

So, while you're right, I don't think it's an entirely useless comparison; it indicates there could be something here, which could then be used to state a falsifiable hypothesis, something like: "Similar cohort drivers would be safer in <one of these ten models> than in a Tesla.".

At a minimum it indicates that Tesla's numbers do not seem exceptionally safe, even if you can't say they're less safe than comparable competitors either.

Given the low number of incidents, you probably want to be looking at something other that fatalities to get a large enough pool of incidents to reduce the impact pure chance; at least in addition to actual fatalities. It's not going to be an easy comparison, that's for sure.

I'm not sure how much of that is about the car versus the driver. Volvo drivers are some of the most slow, excessively cautious, annoying (to others) drivers on the road.

> Quite. Tesla is deliberately comparing their modern vehicle design & wealthy driver base (statistically one of the safest cohorts) with the entirety of the U.S. driving population. This is bad statistics: We expect Tesla vehicles to have far fewer accidents than the mean US vehicle per mile, because Tesla's are expensive modern vehicles with (relatively) wealthy drivers who can afford to keep them maintained & will themselves be in better health than the mean US driver.

You're going to have to explain to me how the wealth of a driver has any bearing on their ability to pilot a self-driving car.

Statistically if you're wealthy enough to own a Tesla, you're also way less likely to have to drive when tired/under the influence for example.

All sorts of constraints are correlated with wealth that put you in way more risky situations if you're poorer.

Do you have anything to back this claim? It seems to not make sense.

For example: https://academic.oup.com/alcalc/article/46/6/721/129644

It's not exactly controversial to claim that lower income correlates to crime though.

There's lots of different kinds of crime, each very unique in its circumstances. Just try comparing the stats for securities fraud vs. drug possession, for example.

DUI is a tough one, people of all socioeconomic backgrounds like to get drunk, and people of all socioeconomic backgrounds like to drive. Off the cuff, I might guess that the numbers would show DUI to be the most egalitarian crime, actually.

How about you look at facts before spewing PC-motivated 'guesses', like http://www.thelancet.com/journals/lanpub/article/PIIS2468-26... and http://injuryprevention.bmj.com/content/8/2/111 - both from the first page of my first google search, I'm sure it's trivial to find many more.

Firstly, wealthy drivers can afford to keep their cars in better shape and can afford better cars in the first place. Neither of these factors depend on self-driving technology.

Secondly, Tesla themselves makes it clear that their self-driving can't be relied upon to driver the car unsupervised - the driver has to be paying attention at all times in case the Tesla self-driving code decides to do something catastrophically stupid. Hence the same factors that correlate with driver attentiveness in the ordinary population will also affect the Tesla population. Wealth correlates with health correlates with better reaction times & general fitness.

The Tesla autopilot tends to turn itself off precisely in situations where driving is difficult and complex - which in turn implies that these human factors will remain just as important as they are in the general driving population.

(Of course it may simply be that the reason wealthy people have a lower death rate on the roads per mile is that they can afford cars that are safer in crashes; the above is my supposition, not evidentially backed.)

What I'm looking for really is anything specific to auto-pilot technology to suggest that the safety benefits won't also be met, or exceeded, when extrapolated across the whole population.

All other factors taken into account, driving has become much, much safer in recent years, and is now safer than it ever was. When you start to use technology to eliminate driver errors in that self-same error-prone cohort, there's no reason to think that the safety dividend won't be even more pronounced than it is today.

If you want to link crash rates to wealth, you would also have to account for countries like Germany, that has a similar median wealth figure to the US, but a vastly lower motorway death rate [1]

It seems you can't both say 1) Tesla drivers are less likely to have accidents because they are wealthy and error-prone, and 2) Extending self-driving technology that eliminates basic errors to less-wealthy and/or more error-prone drivers won't have a net benefit on driver safety.

[1] https://www.sciencedirect.com/science/article/pii/S240584401...

I’m suggesting that Tesla drivers are both relatively wealthy & /less/ error prone!

Clearly /if/ Tesla self-driving code is safer than the average driver then extending that to the general driving population would be a net good. But if the apparent safety is an artefact of the statistics due to Tesla cars themselves being intrinsically safer just as practically every other modern vehicle is a safer car than one built 20 years ago then extending self-driving to every vehicle might not be a net safety benefit

Likewise, if the driver cohort is different (and we know that it is different) from the general population & that cohort is required to drive the vehicle precisely at the times that the self-driving code can't cope which are probably the times when driving places the most demand on the driver, then that different in driver cohort matters: the statistical difference in death rates could easily be a result of differences in driver populations.

You can't tell from the data as given & the fact that Tesla keeps repeating the aggregate figures in their PR is suspicious frankly.

(NB. that paper you link simply underlines my point - In both countries, the lowest wealth quartile has a vastly greater death rate on the roads than the rest of the population. It doesn't matter whether Germany is worse or better than the US. If Tesla draws disproportionately from higher income deciles, then we should a priori expect them to have a far lower death rate than the mean. Quoting the mean as a benchmark to compare the Tesla death rate against is outright misleading.)

>Wealth correlates with health correlates with better reaction times & general fitness

I agree with your general point, but you can't just join up 2 correlations like that, that isn't always going to work when there are other confounding factors (and there always are). Wealth also correlates with age which anticorrelates with reaction times and general fitness.

Agreed, hence my parenthetical note at the bottom of the comment.

According to insurance adjusters, age correlates with lower accident rates.

And having a hotmail emailaddress with higher accident rates.

Lower accident rates, or a lower percentage of injury or fatal accidents? I find it hard to believe 80 year olds are getting into fewer accidents than 40 year olds unless it’s because they drive less frequently.

Which would be fine for an insurer of course. Heck, even if they got into more incidents, but the incidents were cheaper, that would be fine for an insurer too.

And not all markets are equally competitive. Just because insurance rates differ doesn't mean risks necessarily do the same way. And let's not forget that while insurers are professional risk assessors that doesn't mean they're infallible either, or immune to other distortions (e.g. pass-the-buck subprime mortgage kind of issues).

In case anyone's wondering, here's a simple example of 3 variables so that the first is positively correlated with the second, the second positively correlated with the third, but the first and the third have a negative correlation: simply take 3 independent standard normal variables, X, Y, Z, and define our 3 variables to be u=X+Y, v=Y+Z, w=Z-X. Then correl(u,v)=0.5, correl(v,w)=0.5 and correl(u,w)=-0.5.

> Wealth correlates with health correlates with better reaction times & general fitness.

But at some point between wealthy and rich the person hires a (non-wealthy) driver and now they're back to square one.

Wealthy people can better control the quality of the driver.

Wealth of driver - converts to potential good lawyers per damage per crash converts to fear of god in Tesla software departments management converts to actual safety practices being followed and corners not being cut?

That's an awfully long risk compensation feedback loop to close! It sounds like a plausible motivation for Tesla to care more about the quality of their self-driving code than they otherwise might, but it would be a bit difficult to run the counter factual experiment to actually prove it one way or the other :)

Hell, wealth correlates with age. How many 16-20 year olds are driving Tesla’s?

> We expect Tesla vehicles to have far fewer accidents than the mean US vehicle per mile,

We do? Is there a cite for that? It seems to me that driver of new luxury vehicles are disproportionately aggressive and unsafe, actually. Has this been studied?

If your going to make an argument for bad statistics, you need to practice good science yourself.

The simplest argument here is that Tesla's are new vehicles, the total US fleet average age is much older than any Tesla on the road today and driver/passenger fatalities tend to be higher in older cars because they have fewer safety features.

Yeah yeah, I get the argument. The point was that you can't ding Tesla for "bad statistics" by using an argument that asserts "tend to be" without evidence.

In fact I personally doubt (i.e. I have no evidence and am not actually making this argument because it would be "bad statistics") this is true if you compare Tesla's to the BMWs being sold into the same demographic. Those "safe new vehicles" you're positing include a whole lot of crossovers and minivans driven in much safer ways.

Lots of data here:

http://www.iihs.org/iihs/sr/statusreport/article/42/4/1

higher in older vehicles

This seems intuitively that it would be true, but are there any stats on car age and accident frequency? Also where is the line on old? Vehicles made in the last 5-10 years will share most safety features, with all but a small minority of newer cars having some cutting edge. I'll admit I had been thinking of fleet age from the state of maintenance, but safety feature is a fascinating angle as well.. And the article you link in another comment was a fascinating read too, thanks!

For one there is the presence/absence of safety features (that one is obvious).

Then there is the actual design of the vehicles, cars that were made in the last couple of years fare much better in collisions with cars that are older because they are a lot stronger, and are better able to channel the energy of an impact to areas away from the passengers. Many newer vehicles will for instance shift the engine under the passenger compartment in a frontal impact.

Older airbags had 'best before' dates associated with the igniters being somewhat unstable and that reduced the certainty of them going off as they got older.

Then, finally, the bodies themselves, once cars start to rust you can patch them up but they won't be the same afterwards. There is quite a bit of redundancy in the body but it definitely doesn't help if there is significant corrosion.

>> You're going to have to explain to me how the wealth of a driver has any bearing on their ability to pilot a self-driving car.

It doesn't. But autopilot is only supposed to be used on the highway. Not in New York City. Not on dirt roads. On the highway. That's the easiest place to make a system like that work - adaptive cruise control and a simple lane follower go a long way there (I hope Tesla has something a lot more advanced than that). To compare stats on their limited use autopilot to the whole country is truly disingenuous.

Wealthy people tend to be older, which directly correlates to less impulsive or stupid behavior on the road.

People who buy a super expensive car have more to lose than some dude in a 10 year old hoopty.

Maintenance standards vary. We can presume that big brother Tesla maintained the car as they would have castigated the driver for not doing so. They are also the only game in town.

For the same reason those drivers will pay less for car insurance for almost any car they buy: they are actuarially safer drivers.

> You're going to have to explain to me how the wealth of a driver has any bearing on their ability to pilot a self-driving car.

One important nit: it isn't self driving. It's sometimes self driving.

About cohorts: how about this: young (thus inexperienced and more careless) drivers generally can't afford Teslas. So yes, wealth is an indicator for how accident prone someone might be.

Do you expect 370% above average for any cohort? That's amazingly amazing. That must mean that some other cohorts are correspondingly far below the average, what are they?

Thank you. Do people really want Tesla to use another company's failure / worse #'s via statistics in their review of this accident?

At least they are responsive, seemingly open, and seemingly empathetic. They are obviously handling things in a business first manner as corporations will but it hardly feels as inhuman as people seem to be making it out to be.

My question is honest. I found GP's assertion and its implication astonishing, but if that's how it is...

You forget the key rule: "Move fast and break things", including people and expectations of human decency

There's always someone who expects more. If you make a doodah that reduces (name something bad) by 30% someone will expect 40%, if it improves (name something good) by 50% someone will expect 60%, and by the way it costs too much, whatever it costs.

Name your expectations up front, please.

The expectation is simple: value human life. Don't try to do beta tests and simultaneously market it as if it's production quality. Accidents happen, but own up to them; don't immediately and fervently insist that you were not at fault before the investigation even starts.

It may be easy to casually update software and break builds, but when human lives are at risk a healthy dose of conservatism is appropriate.

But you're right -- it's probably unfair to expect a SV company to prioritize anything over profits (we saw that with Facebook, right?)

I thought about this over the weekend. About that you want, concretely.

Tesla says their cars have drive past that spot 85000 times so far. So you may be asking for some testing that's better at uncovering problems than 85000 whitenoise test instantiations are. Or you may be asking to test, and test, and improve until the software meets the developers' intentions: If their intention is to improve safety by 40% (or 400%, or 40000%) then you may be saying they shouldn't release a version that's improved over the baseline by 20% (or 200%, or 2000%) because that still doesn't realise the design.

Seems a bit absolutist to me. You could equally well make the opposite absolutist argument that having inhouse software that's better than the current released is putting people's lives in danger by inaction/negligence/whatever. That fixing a bug or improving a feature confers a moral obligation to release immediately, lest people's life be put in danger which would be avoided by the new code.

Less rich people, less healthy, driving smaller, older, less well maintained, less safely designed cars in more dangerous/remote locations. That’s a bunch of factors contributing to safety or lack thereof. Vehicle age alone is a huge factor in safety. Even the likelihood the driver and passengers are wearing a seatbelt is likely vastly different between Teslas vs all cars on the road. All those factors add up.

But that’s not even the point. The point is that the numbers Tesla gave are meaningless. Tesla’s stats would only be meaningful if you compared them to similarly sized/priced cars of the same age cohort with the same market segment. The fact that Tesla compares its luxury high-priced large sedans to all cars and not its own segment is itself very telling. They have those stats, and they would use them if they were compelling.

I've heard that before and it still sounds puzzling... Elon Musk has an ego the size of Mars, right? So he wants to sell autopilot-equipped cars to everyone, and his basis for comparison is everyone.

(Last posting from me today.)

Have you seen some of the drivers out there? Next time you’re in a parking lot take note of all the cars in terrible condition with scrapes and dings all over.

This is by far the most important point in this discussion. According to this article (https://www.greencarreports.com/news/1107109_teslas-own-numb...), Tesla's statistics are deliberately wrong, and Tesla compared its Autopilot crash rate to the overall U.S. traffic fatality rate—which includes bicyclists, pedestrians, buses, and 18-wheelers.

You could argue that Tesla should be compared to luxury vehicles (which have more safety features), but even if you just compare their crash rate against U.S. drivers of cars and light trucks, the Tesla Autopilot driver fatality rate is almost four times higher than typical passenger vehicles...

They should (on closed tracks) test self driving cars with drunk and deliberately stupid drivers. Do they do this? Because if you mass market such a car you will get people using it to drive themselves home drunk.

They can be far more specific: just tell us how Model S drivers without (or not using) Autopilot do compared to those with and using it. This would control almost every variable except Autopilot. I find the fact that they don't release this number as extremely damning.

Somebody has tried to do this [1]. It would be nice to see this analysis from Tesla themselves. As you say, we can probably guess why we haven't.

[1]: https://www.greencarreports.com/news/1107109_teslas-own-numb...

Reply on news.ycombinator.com