Tesla Gets Carried Away Hyping Crash Test Results, Has Hand Slapped By Feds | Hacker News Reader

Tesla Gets Carried Away Hyping Crash Test Results, Has Hand Slapped By Feds

forbes.com

75 pointsw1ntermute70 comments

It says as much in the official Tesla press release:

NHTSA does not publish a star rating above 5, however safety levels better than 5 stars are captured in the overall Vehicle Safety Score (VSS) provided to manufacturers, where the Model S achieved a new combined record of 5.4 stars.

You can't blame Tesla for taking that number and running with it, given that (1) modeling car safety from two seaters to SUVs on a range from 1 to 5 stars is ridiculous and (2) the star systems suffer from the same problems as contemporary game and movie reviews: 99% of all evaluated products land between 3 and 5 stars, where 3 then becomes the new horrible, compressing the already low entropy of a star value further.

Nothing wrong with the 5 star rating system as far as it goes... the marketing 'usefulness' of a measure is not the be all and end all, in that regards it is very different from video game and movie reviews.

This makes me wonder, do they adjust the rating system over time, and if so, are they doing it frequently enough? You can buy five star cars today, and I assume you could buy 5-star cars 15 years ago, but 5 star cars from 15 years ago wouldn't get rated as highly today, right?

Maybe they need to be more aggressive with their adjustment so that the minimum expected really is 1 star.

Safety ratings are adjusted annually, so a car that was rated 5-stars a decade ago would probably be lucky to get 2 stars today.

Assuming it could even be sold, if it were lacking now-mandatory equipment (tire-pressure sensors, etc).

The VSS is not a star score though. It looks like Tesla got a 0.43 VSS. Tesla was the one who converted that to a 5.4, by making up a methodology, and their wording on that is definitely a bit unclear.

I demand p-values!

sum(p(accident type) * p(estimated of surviving corresponding safety test))

Nitpick: Those things are not p-values.

P-values aren't probabilities you should multiply or add.

At best you're computing odds according to the null hypothesis (probably not what you indented). At worst it's as meaningless as adding meters to seconds.

They have those. Maybe not exactly in the format you specify, but there are statistics for various star ratings and accident survivability.

> (2) the star systems suffer from the same problems as contemporary game and movie reviews: 99% of all evaluated products land between 3 and 5 stars, where 3 then becomes the new horrible, compressing the already low entropy of a star value further.

That's not a bad thing. We wanted safer cars. We started measuring safety, and publishing the measurements. As a result, safety improved. Working As Intended.

The threshold for each star is increased each year, based on the available safety features and safety performance of cars. A car that received 5 stars under the 2000 rating system would receive only 2 stars today because it lacks some of the now-required safety features.

3 is the new horrible simply because that is the lowest that a commercially viable car can go--Chevy and Subaru have released sub-3 cars in the past and suffered for it with poor sales. They and others learned that consumers wouldn't buy cars that were rated below-average safety-wise and have vastly increased R&D spending on safety features. That's a good thing for consumers.

> rated below-average

It better not be a real average if everyone is always above average!

You mean to say poorly rated, not below average, since it's not an average.

Here is how star rating actually works:

For a truly excellent product, the distribution should approximate a power law graph:

    XXXXXXXXXXXXXXXX
    XXXXXX
    XX
    X
    X

If you see something like these:

    XXXXXXX
    X
    X
    X
    X


    X
    X
    X
    X
    XXXXX


    XXXXXXXXXXXXXXXX
    XXXXXX
    XXX
    X
    XXXXXXXX

Then ask yourself if the sample size makes this plausible, in which case the star rating isn't worth much, because there isn't enough data. Ask yourself if it's possible that someone has been gaming the system for or against.

The uppercase gamma-shaped graph is either someone gaming the system or it's a power law graph hidden by aliasing. (In which case, the thing is of godlike quality.)

We should do away with star ratings, and instead just present an aggregate "rating." Genuine customer ratings are valuable, but are almost immediately gamed by cheaters. If I were Apple or Amazon, I would build stealth captchas and "honeypot" traps for ratings manipulation, and figure those into the rating. Be sure to separate detection from consequences by a random time interval of at least a few days. This makes the "development cost" for the cheaters much higher.

The problem with star ratings is obviously what PG refers to as "schlepping". Is anyone tackling this already?

EDIT: I would surmise that AirBNB is somewhat tackling this problem already.

I really like Tesla and want them to succeed, but they've got some weird ideas about good PR. The cost calculator that factored in the cost of pump time at your billable rate; the over-the-top posture against the NY Times reviewer who discovered that the battery heater could draw enough power to affect range; and now this, where they made up their own methodology to convert VSS into stars and found themselves to be the best. (I'm not even considering the weird stuff like Elon Musk offering publicly to consult on Boeing's batteries, although I think that points to the source.) Aren't the "best car ever made" reviews enough?

You left out Tesla suing television show Top Gear for libel. (Which they lost despite the UK's infamously plaintiff-friendly libel laws.)

I think that may have been less about winning the case, and more about getting the word out that the 'test' Top Gear did was edited in a way that maximized humor vs. being 100% accurate.

Sure, and that's probably a fair point. They should have said exactly that. Instead they implied the show maliciously falsified a review and then spent months (years?) unsuccessfully litigating them.

Right, and Tesla lost because Top Gear argued (successfully) in court that what they aired was not a review but a comedy show. And being that the review was not a review, it could not be a malicious review. That's very thin ice to be standing on.

I really think you are mistaken. Do you have a source that says that's why the case was dismissed?

The Top Gear review was essentially accurate, at least IMHO.

Tesla argued in court that it was unfair to mention the car's disappointing range on the test track because most people won't be driving on a test track. A strange argument, especially for a sports car. Note that they did not argue that the range number cited was wrong -- just that it might confuse potential buyers.

The judge ruled that: "no reasonable person watching the programme could understand that the performance on the track is capable of direct comparison with the performance, including the range, on the road"

I thought the problem was Top Gear showed them pushing the Tesla, as if it had run out of juice, when in fact it still had battery and could have driven under its own power?

part of the issue is how quick people are to judge, combined with the particular historical perception of electric cars. Tesla has to be very aggressive about the image of its vehicles, otherwise the prevailing prejudices about electric cars in general will override the image of its own cars regardless of how awesome they may actually be

Their vehicle safety score was 0.42, the closest one was 0.50. That's a big difference. The lowest with a 5 star rating was 0.66 and the average was 0.60. That's totally something they should be allowed to brag about, but they should have just used those numbers or clearly explained their methodology. I think they were just trying to show off that number in a familiar way, and I can't blame them too much for that.

Lower VSS is better? I wouldn't want to try and explain that in marketing.

> the over-the-top posture against the NY Times reviewer who discovered that the battery heater could draw enough power to affect range

Actually he claimed to have the heater keeping the cabin at near-freezing temperatures, and the cabin temperature graph captured from the car strongly disputed that. Going over-the-top at the guy was all they could do after he had made so many different false claims. He was probably right with some of the claims, and the proof of him circling around in the parking lot wasn't as open-shut as initially presented, but when you are hit by that many misrepresentations at once you have to fight back fast.

You mean the reporter who drove around in circles trying to kill the car off and lied about the heater settings he used, also in an attempt to kill the car? I think you missed the whole week there where the actual drive data proved he was a fraud.

I don't like that reviewer, because he's evidently a very sloppy reviewer with a pretty poor understanding of physics and mechanics involved. However, I no longer think he was gaming the system deliberately. Assume incompetence before malice.

He may very well have been incompetent, but at the very least he was dishonest too. He put a serious bias on what he "reported" in his article. It may have been a malicious attempt to kill the car for a better story, or it may have just been an attempt to cover up his own incompetence, but either way, it was a pretty bad article and deserved to be trashed.

He drove around a parking lot once, searching for the charging station.

"The agency has dominion over any part of a vehicle sold in the United States, from the headlamps to the owner’s manual, but can’t regulate the speech of a corporation or its employees."

True, but the agency gives legal cover. When there is a question in court as to whether the safety systems performed "as advertised", the NHTSA is free to say that Tesla didn't follow the advertising guidelines, and therefore cannot comment on the performance of the safety system.

Drug companies can also advertise drugs outside the FDA guidelines, and would win a court challenge on 1st Amendment grounds. None do it because it would expose them to unbounded liability.

>Drug companies can also advertise drugs outside the FDA guidelines, and would win a court challenge on 1st Amendment grounds. None do it because it would expose them to unbounded liability.

No, no, no. This is utterly and confirmably false. It is illegal under the Federal Food, Drug and Cosmetic Act to advertise a prescription drug for a use that is not FDA approved, and it is within the FDA's mandate to recommend investigation to the Department of Justice in such circumstances. The FTC is authorized to oversee advertising for over-the-counter drugs under the Wheeler-Lea Act. As a recent example, Pfizer agreed to settle with the DoJ earlier this summer for half a billion to end an investigation brought under these very circumstances[1].

[1] http://www.newstatesman.com/business/business/2013/07/pfizer...

Jesus man. You seem awfully certain. Here is a ruling from December that affirms 1st amendment rights to market off label uses:

http://www.reedsmith.com/files/uploads/DrugDeviceLawBlog/Car...

" Caronia argues that he was convicted for his speech -- for promoting an FDA-approved drug for off-label use -- in violation of his right of free speech under the First Amendment. We agree. Accordingly, -3- we vacate the judgment of conviction and remand the case to the district court. "

It is uncertain whether marketing off label is protected by the 1st amendment. The courts don't all agree.

> The courts don't all agree.

The one case you cite is very fresh precedent in a single Circuit, and it's highly controversial. No other Federal Circuit has applied First Amendment standards to FDA misbranding regulations before. Even within the Second Circuit (three states), is it still illegal under law to misbrand a drug in marketing materials. The decision in Cariona hinges on the question of whether "misbranding" and "promotion of off-label use" are different narrowly defined legal scenarios. The Second Circuit was concerned that, in the latter scenario, doctors may be legally deterred from promoting off-label use, which is a common practice in medicine and research. You should not take away from this case that drug marketing is now a First Amendment free-for-all. Yes, I am awfully certain.

This is an interesting case, however its important to point out that the legal debate around this is ongoing. This ruling only applies to the 2nd Circuit at this point. It's looking like a Supreme Court case will be necessary to ultimately decided this issue, given the murkiness around it.

Until then, however, off-label drug marketing is not something companies get away with without a substantial legal battle (often not worth it), and the FDA has so far collected billions of dollars in settlements and fines through successful off-label drug marketing suits.

I've been out of the industry for a while but I spent 5 years working on regulatory projects in the MD&D sector.

We never advertised off-label. Whether it's legal or not, I do recall you'd get slapped by the FDA for lack of controls. We had many cases where doctors would use a product off-label and it'd come back to us in the form of a complaint or defect (product comes back as defective, we investigate and find out the doctor was using the product for an off-label procedure it wasn't designed for and would fail, hence the 'defect'). At which point we'd have to proactively train doctors to not do this and other corrective action measures would have to be put in place. If upon investigating we find this would be a good use of the product, we'd have to go through another round of FDA fun (sometimes clinicals and how much testing depended on some factors, definitely rounds of internal testing and implement new QA tests and procedures to ensure the product can work for the new use and have proper fault tolerances as well as new labeling and training, etc).

Now, do companies try to skirt around this? Sure. Part of my job at the time was to make sure we didn't. FDA Warning letters are no joke in this industry, it can literally put you out of business.

This is an interesting 1st Amendment discussion (relating to the article...I am not touching the separate FDA/drug issue).

The most analogous case that comes to mind, is the recent California small claims case against Honda.

Facts: According to the EPA (who independently tests vehicles fuel efficiency) the Prius gets an average of 51 MPG. The Plaintiff claimed she only got an average of approximately 30 MPG. The Plaintiff won the lawsuit against Honda for fraudulent misrepresentation winning $10K the small claims limit in the jurisdiction. However, one of the most interesting arguments (and legitimate) was that while Honda does advertise the 51 MPG, that Honda could not be fraudulently misrepresenting the MPG, because Honda had no control over the testing or findings handled by the EPA, and going further Honda does not have a right (1st amendment) to contradict the "EPA sticker". Ergo according to the argument Honda should not be liable, rather the EPA - nevertheless the Court found Honda liable for fraudulent misrepresentation.

Assuming this law is on point and controlling, for example, if the Tesla were to under perform in a real crash and Tesla was sued, Tesla may be liable despite having no control over the NHTSA test or findings. Going further, if the dicta from the Honda case is also on point, if Tesla's internal crash tests contradict NHTSA's ratings, Tesla would not have a 1st Amendment right to contradict the NHTSA findings publicly.

Note: That SCOTUS makes a distinction between "commercial speech" and other forms of speech, and the government has a lot more power to regulate commercial speech than say political speech.

Are you referring to a suit against Toyota or a Honda hybrid vehicle? Prius is manufactured by Toyota, not sure why Honda would be involved.

Nice catch...the suit was against Honda. I must have subconsciously invoked Prius as a matter of bias (not in a negative capacity, simply as the dominate hybrid in my mind subjectively)

It just seems like the NHTSA is just reiterating that there's no official higher than 5 star rating to protect themselves, rather than scolding Tesla. Tesla did say that a rating higher than 5 stars does not exist, and that the extra high rating should be taken with a grain of salt seemed implied to me, but I guess they could have been clearer. The blog post was very in depth, so it's not like they were misleading people with their wording.

Yeah. This is a non-story; it makes a good headline but no one has been slapped. Tesla just noted that they received some papers which noted that for the internal purposes of the ranking, it got 5.4 stars. Then the NHTSA made a clarification that the official ranking actually doesn't go past 5 stars.

It's good for Tesla shareholders that Forbes decided to run with this story, though. Free advertising. There is no way anyone will ever perceive a fight about whether the car should have 5 or 5.4 stars as negative.

Does it really hurt consumers if carmakers are bragging about doing well on tests?

I'd argue that it might: NHTSA wants to encourage automakers to build safer cars. One way to do that is to provide them with detailed information on the results ("scores") of its tests, so that automakers can benchmark their engineering against each other and tune individual safety features as needed. Another way is to aggregate those results for each model and publish a consumer-friendly figure that automakers are motivated to be able to advertise as a "stamp of approval".

What they don't want is automakers getting into NHTSA test "hot-rodding" games, focusing their efforts on topping the individual tests by some small relative figure in order to claim things like "safest car ever". In an environment like that, NHTSA has to either stop publishing detailed results, or keep the mechanics of their testing secret in an attempt to prevent "benchmark gaming" of the kind we see all the time in technology. That sounds like an NHTSA that is competing with automakers rather than working with them, and I suspect that is how consumers could be ultimately hurt.

This is nonsense: "can’t regulate the speech of a corporation or its employees"

Entities have rights over how they are used in promotions, i.e., the right of publicity / personality rights http://en.wikipedia.org/wiki/Personality_rights

For example, Consumer Reports No Commercial Use Policy - http://www.consumerreports.org/cro/about-us/no-commercial-us...

I wonder how many other cars got more than five stars by Tesla's methodology? From the graph, it looks like some have very close ratings.

And clearly, the goal here is to stop every manufacturer from just coming up with their own rules for interpreting the score that give them a different rating from the one that the NHTSA gives. That's definitely worthwhile.

It was easy to see that coming once the media story took off the way it did. But this just generates more coverage of Tesla's results.

I doubt that anyone at Tesla is feeling the least bit chastised. Musk is certainly not the type of person who is afraid to generate a little controversy.

Official announcements on http://www.safercar.gov/ (image) and http://www.nhtsa.gov/Misc/HeadLines (first headline)

So a problem with precision and judgment, but not integrity?

and integrity:

> NHTSA does not rate vehicles beyond 5 stars and does not rank or order vehicles within the star rating categories.

So the following statement is flat-out untrue:

> SETS NEW NHTSA VEHICLE SAFETY SCORE RECORD

If NHTSA does not rate vehicles beyond 5 stars, the maximum safety score is 5 and thus 5.4 is irrelevant

I think if the headline was "sets new vehicle safety score record", dropping the NHTSA acronym, then it would be fine.

NHTSA doesn't PUBLISH ratings above 5 stars. Regardless, the 5.4 IS relevant.

The fact that the NHTSA formula gives a score of 5.4 doesn't mean that NHTSA recognizes the car as superior to those with a technical score of 5.3. To put it differently, if a class gives a grade of A,B,C,D,or F, having a numeric score of 99 versus 98 is somewhat irrelevant if the ultimate grade is A in both cases.

If they really did want to make a distinction, they'd introduce a new scoring metric that does allow for those differences

If you were in school and had a bet with your roommate about final grades, you wouldn't call off the bet as a draw if you both got A's but you got a higher numeric score than he.

If you had a bet with your roommate about final grades, you would look at the transcript at the end of the semester. And that transcript most likely says A for both of you, so it would be a draw.

I'd jump on the web and see the grades posted by the professor on the course website actually. I always got numeric grades at the end of the semester unless I waited long enough that the course website was nuked, or unless that professor was lazy.

You could also just go ask the professor at their office for your grade, but using their site is easier.

For a bet like that, it doesn't matter if the score difference results from random chance and not any underlying difference in skill or ability. The NHTSA is in the business of assessing car safety though, and it makes little sense for tnem to rank cars within a category if they don't think the difference is actually meaningful.

But having a numeric score of 59 versus 9 is relevant, despite them both being F. Likewise, 89 versus 90. NHTSA can do whatever they want; I can say it is just a silly metric that is about as good as gaming reviews, as another comment points out. A 5.4 score is still relevant; why would they throw away precision?

The NHTSA probably doesn't think that a safety score of 5.3 versus 5.4 has any sort of meaning. If the set of accidents under which you would survive unscathed in a car with score 5.4 but would be injured in a car with score 5.3 has measure 0 (or negligible), then there is no real difference in terms of safety (even if the scores are different)

OTOH, the whole rating system is about taking real tests and progressively throwing away levels of precision. Like combining front and rear and side impact into a single score.

The Tesla blog post makes it abundantly clear that the 5.4 is the report the manufacturer receives (not the issued rating which is capped at 5). Why does it stop at 5 if you can still die in those cars?

I don't think it does make that clear. It says "safety levels better than 5 stars are captured in the overall Vehicle Safety Score (VSS) provided to manufacturers" but the VSS value for the Tesla was 0.43 according to the blog post. So where did the 5.4 come from? Extrapolation?

You're right - they didn't make it abundantly clear that the VSS report was 0.43. Regardless, the conversion they made isn't insane when you look at the other VS scores. Plus - watch the videos of other 5 star rated vehicle versus the Model S. It's like the comparison between a 1980s car and one from 2013.

NHTSA Six Stars: The car is a mobile fortress.

All I meant was that if 5 stars isn't perfect why is that the scale they're using? Obviously the range of what a 5 star rating represents could mean the difference between life and death in a serious collision.

I am clearly agreeing with you. A mobile fortress is far more perfect in terms of safety than any consumer car on the road.

It might not move fast, but you need military-grade hardware or construction equipment to make a dent.

ObHeadlineHypeComplaint: that's a "hand slap"?

I take it from the article, nothing but a statement from the safety agency asking "car manufactures" (not even mentioning Tesla) to not misrepresent their data to the public.

Better to ask for forgiveness than permission.

why is shit like this on the HN frontpage?

Because it's getting upvotes, mostly.

HN isn't just about software/engineering, it's also about entrepreneurship.

Tesla is a startup with cool technology. They're taking on a market where startups never succeed and are navigating significant challenges regarding new technology, consumer bias, competitors, regulatory issues, business model, press relations, and more. Some of us find it fascinating and enlightening to watch them take this journey.

Edit: Reworded.

The Tesla Motors post that heralded this news (https://news.ycombinator.com/item?id=6240862) was on the front page for nearly a full day and garned 794 upvotes. HN readers are clearly intrigued by TM

Because Elon Musk is god here and worshipped.

Reply on news.ycombinator.com