Hacker News Reader

Comment by heavyset_go | Hacker News Reader

> On Thursday, NHTSA said it had discovered in 16 separate instances when this occurred that Autopilot “aborted vehicle control less than one second prior to the first impact,” suggesting the driver was not prepared to assume full control over the vehicle.

> CEO Elon Musk has often claimed that accidents cannot be the fault of the company, as data it extracted invariably showed Autopilot was not active in the moment of the collision.

> While anything that might indicate the system was designed to shut off when it sensed an imminent accident might damage Tesla’s image, legally the company would be a difficult target.

View on HN

So what happens is

1. Speed straight towards your doom.

2. Give back control a moment before collision, when it realizes it fucked up.

3. "It was the driver's fault"

Exactly as predicted by many, many skeptics. Of course that's what his statement meant.

If you read the actual report, you will see that NHTSA actually acknowledges that autopilot makes things safer overall, and they agree that crashes are due to misuse of the software. They highlight that on average, when there was a crash, the driver should have been able to recognize the hazard ~8 seconds in advance.

The challenge of course is what level of misuse is to be expected (especially once it becomes more widely available), and if using the software with normal levels of misuse results in an overall safer level of driving than without it.

It appears to be so: misuse is included in the overall safety assessment.

But this is exactly the position of NHTSA in the report. They say that if Tesla can reasonably do something to mitigate misuse, then they should. That’s what this is about.

My point is:

1) This is currently limited to a pretty restricted market (people who drive Teslas, which is a small subset of car drivers). When used more widely by a wider set of people, it may have a worse track record.

2) It isn't clear that such misuse has reasonable mitigations that will continue to make it attractive to use.

But it's all speculation at this point.

> If you read the actual report, you will see that NHTSA actually acknowledges that autopilot makes things safer overall

Where does it say that in the report?

> and they agree that crashes are due to misuse of the software

I also do not see this in the report other than acknowledging that is Tesla's stance. In fact, they explicitly reject that as a defense, to quote:

"A driver’s use or misuse of vehicle components, or operation of a vehicle in an unintended manner does not necessarily preclude a system defect. This is particularly the case if the driver behavior in question is foreseeable in light of the system’s design or operation."

> They highlight that on average, when there was a crash, the driver should have been able to recognize the hazard ~8 seconds in advance

That's simply not what the report says. Here's the passage you're referencing:

"Where incident video was available, the approach to the first responder scene would have been visible to the driver an average of 8 seconds leading up to impact."

_Except_ you're leaving out some extremely important context.

1. This sentence is part of the expository section of the report, it's in no way "highlighted".

2. This is referring only to 11 specific collisions in which Teslas struck first responders, not the other 191 collisions they examined.

I've seen some other things you've said in this thread so I feel like I should address those as well.

> this is only autopilot not FSD.

To quote the report:

"Each of these crashes involved a report of a Tesla vehicle operating one of its Autopilot versions (Autopilot or Full-Self Driving, or associated Tesla features such as Traffic-Aware Cruise Control, Autosteer, Navigate on Autopilot, and Auto Lane Change)."

> this will be fixed by an over the airwaves update.

I see nowhere in the report that says this. In fact as far as I can tell from reading it the report says they're upgrading and escalating their investigation because of the seriousness of the concerns.

Here's the report's conclusion in its entirety:

"Accordingly, PE21-020 is upgraded to an Engineering Analysis to extend the existing crash analysis, evaluate additional data sets, perform vehicle evaluations, and to explore the degree to which Autopilot and associated Tesla systems may exacerbate human factors or behavioral safety risks by undermining the effectiveness of the driver’s supervision. In doing so, NHTSA plans to continue its assessment of vehicle control authority, driver engagement technologies, and related human factors considerations."

Actual report here for those curious: https://static.nhtsa.gov/odi/inv/2022/INOA-EA22002-3184.PDF

And for those who don't here's the best summary I can provide.

* The government received 11 reports of Teslas striking first responders with autopilot engaged.

* After an initial investigation of these reports they deemed them serious enough to open a broader investigation into the technology.

* They found 191 cases where vehicles using the technology crashed unexpectedly.

* Of those 85 were dismissed because of external factors or lack of data (leaving 106).

* Of those "approximately half" could be attributed to drivers not paying attention, not having their hands on the wheel, or not responding to vehicle attention prompts.

* Of the 106 "approximately a quarter" could be attributed to operating the system in environments where it's not fully supported, such as rain, snow, or ice.

* That leaves 37 cases where it appears the cars malfunctioned, both not prompting the driver and not taking preventative action like braking until it was too late to avoid the crash or not at all.

* As a result, the NHTSA is escalating their investigation to do an engineering analysis of these systems looking to better understand the defects.

Plot twist: it's always the driver's fault. Autopilot is glorified cruise control (the technical term is 'Level 2' on the SAE scale). It literally doesn't matter when Autopilot bails out altogether (even if it manages to automatically avert a crash on other occasions), the driver must be fully in control at all times. Would you trust your cruise control to prevent a collision? Of course not, that would be crazy. It's not what it's built for. What a worthless discussion.

The problem is that people can’t actually take over fast enough necessarily. If you’re not actively driving, then you can’t just immediately react.

The NHTSA report says that on average, an attentive driver would have been able to react 8 seconds before the crash. 8 seconds is a lot of time on the highway (where autopilot can be used).

Yet the autopilot system chose to only warn the drivers 1 second before impact, which is basically no time at all to react to anything.

So if those 8 seconds of "autopilot ain't too sure" actually exist, then the car needs to signal that to the driver so the driver can be ready to take over.

The fact it still doesn't do that, speaks volumes about the kind of "progress" Tesla seems to be making, none of it good.

The crashes were not of a kind that Autopilot could have detected earlier. Several involved stationary objects, that Autopilot seems to intentionally ignore because it can't tell apart what's actually in the way of the car from harmless features of the surroundings that will never cause a collision. Which just goes to prove: Autopilot = glorified cruise control. You must never expect it to keep you safe.

> Autopilot = glorified cruise control

Isn't that what the name implies? An autopilot keeps heading and speed.

Sure, but what does that have to do with Autopilot? "Driver gets into a crash because they were spaced out and not actually paying attention to the road" is something that happens all the time, no need for modern ADAS.

Exactly, the most important metric/question is whether it improves overall safety compared to the baseline.

That's the metric Tesla and the self-driving car industry would like to use, but I don't see why it's the most important. Also, it misconstrues 'most important' to mean 'all that's important'.

We could just ban personal vehicles to improve overall safety.

Because “drivers space out when using a system that removes the stimulus of driving that keeps them alert” is not solely the fault of the individual when it’s not possible in real world scenarios to use it safely.

But you're supposed to be actively driving with autopilot.

I use autopilot every now and then, and when I do, I literally keep my hands on the wheel in a driving position, and my foot resting lightly on the pedal.

You by definition can’t be actively driving. You’re just watching. It’s active driving that keeps you alert in the first place.

Then what are you going from Autopilot? How's foot hovering any more comfortable than lightly putting it down? How is hands on wheel in a driving position more comfortable than occasionally turning the wheel?

Seems a tiny gain compared to the dangers of the system.

Very late reply but...what I gain is basically nitpicky adjustments to lane and speed control. I'm extremely active in the sense that I am constantly observing near, medium, and far distances while driving, but I'm just not providing minor steering inputs directly.

I think that I can better observe the conditions if I'm not providing those controls, even though I need to keep a model of autopilot and its abilities running in my head. Having said that, that model is simple - dump out of autopilot as soon as there is any kind of complexity added to the environment.

It seems so weird to me how people so vehemently defend Tesla instead of holding them to higher standards. What do you owe them and why?

Some might also be defending their stock portfolios, 401ks, and future tesla purchases built on tsla price appreciations.

Well when you wrap your identity around someone else's idea, you get weird outputs at the edge cases

> the driver must be fully in control at all times

Yes, that's precisely what the manual said last time I checked. But it's not how Elon Musk has presented it, nor how innumerable Tesla drivers treat it.

Tesla say in the video here the driver is only there because of regulators, not ethical reasons:

https://www.tesla.com/autopilot

This video was from 2018 or so I think and they pretended the driver wasn't needed.

The whole concept is flawed.

We cannot expect humans to be able to pay attention and be able to take over at a moment's notice at all times, simply because that's not how our brains work.

Autopilot is in the danger zone, where it's good enough to make your brain relax, but it's bad enough to require your brain to not do that. So it's fundamentally unsafe.

Cruise control in contrast isn't good enough so your brain will have to pay attention, otherwise you'll crash very quickly.

And this is all made much, much worse by Elon's and Tesla's irresponsible marketing, and the believers who dismiss this as a "worthless discussion".

> We cannot expect humans to be able to pay attention and be able to take over at a moment's notice at all times

And yet, this is what ordinary driving requires already. It's not a new requirement. The whole point of this investigation is to figure out the matter of whether a SAE Level 2 ADAS really makes distracted driving more likely. The jury is still out on that, and even if it does it would be reasonably easy to cope with the problem by tuning the system to alert the driver more frequently, especially ahead of impending danger (such as in low visibility conditions, which are expressly mentioned as a key factor in these crashes).

And yet, this is what ordinary driving requires already

No, it's not. Not at all.

>And yet, this is what ordinary driving requires already.

What? No it isn't. Ordinary driving has the driver controlling the vehicle at all times.

> We cannot expect humans to be able to pay attention and be able to take over at a moment's notice at all times, simply because that's not how our brains work.

Plus, it will always give you the control back in the most difficult situation you can ever encounter as a driver. It's already an emergency, the computer threw a fit and doesn't want to deal with it.

You're on your own, you have 10 seconds, all the best! Better get it right the first time.

> legally the company would be a difficult target

Is this the case?

Look at the Toyota "Acceleration Gate" fiasco. Toyota paid billions(full amount isn't disclosed due to NDAs in hundreds of cases) because of poor software. If Tesla engineers failed to adhere to industry best practices(hard to do when you're using deep learning in your control loop) then they'll likely be liable.

An overview of the Toyota scandal: https://users.ece.cmu.edu/~koopman/pubs/koopman14_toyota_ua_...

I remember reading that at the time, and just looked at the slides again.

A bug which commands wide open throttle also therefore depletes engine vacuum, leading to a lack of power assist if the brakes are pressed and released and pressed again.

Drivers who have been around the block a bit and have some mechanical sympathy would take a moment to realize neutral will resolve the situation. But many other drivers would not realize this.

Although it would not absolve Toyota from responsibility in such a case, I wish driver training required and tested for dealing with a long list of adverse situations within N seconds each.

Reply on news.ycombinator.com