Hacker News Reader

Comment by meepmorp | Hacker News Reader

> The safety agency informed Tesla weeks ago that it was looking into the spate of crashes in which Tesla vehicles operating on Autopilot failed to detect stopped emergency vehicles with flashing lights. The regulator originally said it was looking into 11 such crashes. A 12th occurred on Saturday, when a Model 3 hit a police cruiser that had stopped behind a car that had broken down on an interstate in Orlando, Fla.

12 crashes into vehicles with flashing lights on them doesn't sound like great success from Autopilot, gonna be honest.

View on HN

We just did a 1,500 mile round-trip journey to visit family. Because of the Tesla stuff, I was casually aware of roadside emergency vehicles moreso than usual. We passed 28 police/fire/emergency vehicles that were next to our travel lane.

I saw a number of Tesla vehicles on the road as well, each passing these same emergency vehicles, but not always in the same lane.

By this point, there must be 1M+ incidents of Tesla vehicles passing roadside emergency vehicles in an adjacent lane.

12 crashes is 12 crashes too many, statistically speaking, but it's also a tiny tiny percentage of the total number of cases.

IMO, it is good that there is federal investigation into this. My main concern is that is seems our government agencies at large are not very good when it comes to properly understanding anything technology related, or making relevant decisions/rulings/etc. on these kinds of things.

> IMO, it is good that there is federal investigation into this

> seems our government agencies at large are not very good when it comes to properly understanding anything technology related, or making relevant decisions/rulings/etc. on these kinds of things.

Not sure how you think it's good then.

>I saw a number of Tesla vehicles on the road as well, each passing these same emergency vehicles, but not always in the same lane.

How do you know all of them were on autopilot?

>12 crashes is 12 crashes too many, statistically speaking, but it's also a tiny tiny percentage of the total number of cases.

It's also 12 more than any normal attentive human has gotten into, and an average attentive human driver is the absolute floor for self-driving on public roads unless Tesla is going to go to the lengths GM has to ensure the person behind the wheel is remaining attentive.

>IMO, it is good that there is federal investigation into this. My main concern is that is seems our government agencies at large are not very good when it comes to properly understanding anything technology related, or making relevant decisions/rulings/etc. on these kinds of things.

I think they're good at understanding if Tesla loses complete public confidence in self driving that is a net-loss for humanity and technology, as well as a prime source of political shitstorm when people ask why something wasn't done sooner.

> It’s also 12 than any normal attentive human has gotten into

Are you comparing a fleet of cars driving millions of miles to the performance of one ostensibly good driver? Ok, I guess we’ll need perfect systems, see you after the heat death of the universe.

There's never going to be a perfect comparison when the same AI can drive every car in the fleet at the same time, whereas one human brain can drive at most one car at a time.

Yeah but the issue is at hand is specifically 12 cars. You can put 12 humans in cars and watch them brake before hitting emergency vehicles. If they’re claiming AI is better than humans then it’s important to understand why these 12 are so bad.

I get this argument in abstract, but I don't think it's fair when we're talking about obviously unsafe behaviour that literally no human driver should ever do, except in cases of like, extreme medical distress.

It's about the unknown unknowns. If you have Teslas slamming into stopped emergency vehicles with their lights on, and you don't know why, what else don't we know about their ability to perceive and reason about the world that they're navigating in?

Humans should never do it, but I'd be surprised if it happened less than once a week.

Really? Not steering into a stopped, highly visible emergency vehicle with its lights on?

Humans do a lot of dumb, irresponsible things on the road, but this is not one of them.

It shouldn't even be a question, seriously?

This cop[1] tells a lady not to stand between the vehicles because she will get killed if his patrol car gets hit, and seconds later it happens and he says "just like that".

>this is not one of them.

Yes[2], this[3] is[4] one[5] of[6] them[7].

[1] https://www.youtube.com/watch?v=fZDIwaRWCl0

[2] https://www.youtube.com/watch?v=97D3llj0KSQ

[3] https://www.youtube.com/watch?v=pV5FrWTATDY

[4] https://www.youtube.com/watch?v=loex5U9zD0s

[5] https://www.youtube.com/watch?v=khz9hlpvhko

[6] https://www.youtube.com/watch?v=uyaa6zEHZ2Q

[7] https://www.youtube.com/watch?v=piT4lZBGU4w

This is the reality of the situation and, frankly, should be a top-level comment. I don't really have a horse in this overall race but IMO your factual perspective is spot-on -- this has happened, plenty of times, by actual humans.

Just like anything else, it may not be common (or it may be 'uncommon') but, to say that it never happens, is plainly, clearly and objectively false.

The failure modes of a new driving system are going to be distributed differently than the old system. This is exactly the same flawed argument as "we should ban all human drivers because in the situation where they get drunk they have lots of accidents and self-driving cars don't have that problem".

That said, I suspect there is some systematic issue with Tesla's system and flashing lights, and I bet they are working like demons to identify and fix it already.

which unfortunately also makes them less predictable for other drivers and thus harder to account for/react to in time

Nope, he is simply saying that a human of sound state of mind does not have these accidents.

The authorities are simply viewing Tesla's credibility as undermined. They ask themselves, if it can't do this, how can it get anything right, or there must be many more flaws.

The humans do not like to pass over control to machines unless the machines do it clearly better.

And in this case, the authority does not want to have their credibility shot by looking away from the Tesla issue, it cannot be pushed under the metaphorical carpet anymore, there is a bulge under the carpet visible to anyone who wants to see.

>It's also 12 more than any normal attentive human has gotten into

Can you clarify this please? Are you asserting that prior to the 12 Tesla incidents, there weren't any crashes into stationary emergency vehicles?

No, he's asserting that a normal attentive driver won't crash into a stationary emergency vehicle under remotely normal weather/road conditions. And, personally, I'm not going to trust a system that isn't as good as a normal attentive driver.

Thanks, I'm not sure how I missed that.

>I'm not going to trust a system that isn't as good as a normal attentive driver.

As I understand it, we're not expected to trust the current incarnation of Tesla Autopilot. We're supposed to be ready to intervene in case it screws up.

I'd be interested to know how long the drivers of the 12 Teslas in these accidents had to intervene but didn't (how long was it obvious Autopilot had screwed up before the impact).

My problem with this is that we have little proof that people are any good at this sort of task (i.e. monitoring something without intervening for really long stretches of time).

So if data show that people keep failing at it we have to take a look and decide that it's not a good system, not that the people are at fault.

Even more than that, we actual have multiple studies demonstrating that people are actively bad at this sort of task.

The challenge, which no one has a good answer for, is that if you do a "good" job of automating something, people will zone out. It's just how we're wired.

Does it have to better than an attentive driver or the average driver? I figure if the Tesla has less crashes per mile than the average driver than that's a success?

Even if the accidents a Tesla does get into are ones that a normal attentive driver wouldn't get into, if it avoids just as many accidents that a normal attentive driver would get into isn't that enough?

Well, I'm an attentive driver, so I wouldn't use anything that drives worse than me.

A lot of people probably feel the same.

The average driver thinks that they are a better driver than the average driver right?

While that is true, there are also people who don't drink and drive, don't text while driving, don't drive while overly fatigued, etc. So it's reasonable for those people to not want to use a system that's only as good as an "average" driver.

> So it's reasonable for those people to not want to use a system that's only as good as an "average" driver.

But... no one is forcing anyone to use this system. They elect to. Thus this is the wrong question, but rather if drivers should be given the option to use the system.

If anything it might be reasonable to infer that poor drivers are more likely to use this system, so the standard could plausibly be below that of an average driver while still improving overall safety.

I hate to put it this way, but I think it'll be easier to get people to use autopilot (that performs better than your average driver) than it will be to get people to get gud at driving. You're an attentive driver right? You've probably noticed people lane changing without looking, causing or almost causing collisions. Or jockeying for position fruitlessly. Or being drunk or getting sleepy. Or getting aggro. Back when I was driving a lot I would have been grateful for anything that meant I wouldn't have to drive around people like that. I dunno where I'm going with this exactly, I suppose we've had similar experiences and it makes me a little surprised that you'd be against automated systems bringing the skill floor up.

I would even assert that normal inattentive driver won't crash in stationary emergency vehicle... Or if they do they should permanently have their right to drive removed. I would consider giving attention to flashing lights minimum standard.

Approximately 12 percent of all interstate highway deaths result from shoulder accidents. This means that an estimated 600 people a year are killed—and thousands more are injured—while making emergency stops off the highway.

It’s not just autopilot.

>How do you know all of them were on autopilot?

I don't, but statistically speaking, some of them likely were.

Tesla has shipped about 1.6M units globally, with most of those in the US. Let conservatively estimate half of those stayed in the US, and half of that half have used autopilot frequently enough for a high degree of opportunity of these kinds of crashes. That is still 250,000 vehicles. 12 of those being involved in crashes is a pretty low number, and we also do not know how many scenarios they avoided that a human might have missed.

> It's also 12 more than any normal attentive human has gotten into

What is your definition of "normal attentive human"? There are tons of studies showing how humans suck at long-term tasks that are repetitive or non-engaging (like highway driving). People get distracted all the time while driving and cause accidents. Sure an "attentive" human would not run into a police vehicle, but there are not enough of those on the road most days.

I'm sorry but the general computer called your brain is not like the purposebuilt computer of Autopilot. 12 is too many for a computer who's only job is to literally look at the road and drive.

The standard should probably be “better than human”, not “perfect”.

Maneuvering around congested streets I agree is a judgement call that humans and computers can get wrong. Not hitting stationary objects doesn't take a supercomputer to solve it.

Unless it does.

The NHTSA has been very unclear about how many of these were on autopilot vs cruise control.

> By this point, there must be 1M+ incidents of Tesla vehicles passing roadside emergency vehicles in an adjacent lane.

> 12 crashes is 12 crashes too many, statistically speaking, but it's also a tiny tiny percentage of the total number of cases.

That perspective is little more than spin. Most serious flaws are like that.

Any failure that isn't a "tiny tiny percentage of the total number of cases" is pretty serious. If it was even a small percentage (say 0.01%), Tesla autopilot would be banned from the road, and maybe Elon Musk would go to jail (or at least Tesla would probably get bankrupted) for releasing such a hazardous product.

The fact that Tesla and Elon have been fighting these autopilot errors in completely the wrong way is the primary reason the feds should be involved.

I have been using autopilot in Nissan. If the car were to crash into an ambulance light going off I would not be able to predict it and stop it from happening quickly enough. I just have a whole lot of trust in some basic things the car does.

And this is one of those cases where they should have been able to get this right after the first crash but they didn’t and now the feds have to get involved. A lesson on why we can’t have nice things.

>> The safety agency informed Tesla weeks ago that it was looking into the spate of crashes in which Tesla vehicles operating on Autopilot failed to detect stopped emergency vehicles with flashing lights. The regulator originally said it was looking into 11 such crashes. A 12th occurred on Saturday, when a Model 3 hit a police cruiser that had stopped behind a car that had broken down on an interstate in Orlando, Fla.

> 12 crashes into vehicles with flashing lights on them doesn't sound like great success from Autopilot, gonna be honest.

It's really surprising what people will put up with. IIRC, recently "autopilot" had some regressions in the situations it could handle, and Elon Musk tweeted something to the effect of "Yeah, this version sucks since we're retraining all our NNs."

Like how did something that half-baked ever make it to production in a safety critical system?

> 12 crashes into vehicles with flashing lights on them doesn't sound like great success from Autopilot, gonna be honest.

I'm sorry, but how can you form an opinion so quickly? 12 crashes out of what? Any number? 12 crashes out of 10000000 scenarios sound like a pretty good record, while 12 crashes out of 15 scenarios indeed doesn't sound like a great success.

But what information do you have that the rest of us doesn't?

yes. 12 crashes out of the maximum possible number of encounters.

I still remember the days when the SciFi of the day sold robotic drivers as "they don't get tired, they don't make mistakes, they are relentless in their pursuit".

Instead of that, we got robots that are apparently unable to avoid crashing into large vehicles with flashing lights. Sometimes.

As usual, the future never ceases to disappoint me.

Well if you're basing your expectations of reality on SciFi books than I expect you will be dissapointed quite often.

There's always dystopian scifi as well.

Yes! - This has not failed to disappoint, rather it has exceeded all expectations!

Maybe we just haven't reached that future yet.

I heard it's tomorrow.

Ah yes the 1960s. Where politics and economics were also utopian and dishonest. When your average American job could impossibly support a family of 4. 16 year olds drove dodge chargers and the economy was so strong from all of the various nefarious engineering that you could buy products that were made to last 50 years.

Stories from back then are pretty unrealistic, but that’s true for fiction in general. Dunno why we should point at that in particular.

"they don't get tired, they don't make mistakes, they are relentless in their pursuit".

I believe you're thinking of terminators, not automated drivers:

https://quotegeek.com/quotes-from-movies/the-terminator/1136...

Though the "will not stop, ever...", even for an emergency vehicle, might be relevant.

"I'm sorry, but how can you form an opinion so quickly?"

It doesn't really matter. They're stationary objects; there's absolutely no reason, good or bad, for the algorithm to determine that the correct trajectory change is into the stationary object.

Humans crash into stationary objects because they're lacking attention almost 100% of the time, whether it's because they're half asleep, drunk, texting, or distracted by just about anything else. Autopilot shouldn't have this issue, so there's no reason for it to fail to recognize the obstacles unless it's been poorly trained, and there's absolutely no reason for it to recognize an obstacle as a valid path or destination.

There is absolutely a good reason for adaptove cruise control systems to ignore stationary objects: https://arstechnica.com/cars/2018/06/why-emergency-braking-s...

Is the “good reason” here that they have insufficient sensors?

Which is...linking a Wikipedia article? Doesn't quite compute.

Was a copy / paste failure. Meant to grab this link:

https://arstechnica.com/cars/2018/06/why-emergency-braking-s...

If you mean "the software is too stupid to be useful here, so let's ignore a whole class of issues, potentially with lethal consequences; abracadabra, it's SEP now", I agree. OTOH, this stretches the meaning of "good" a little bit.

(thanks for the link correction)

The reason is "our code is too stupid, because is not an actual intelligence and better sensors are expensive"? This is a bad reason from my point of view.

You claim that you have some super duper AI but it can't detect a truck or a wall from a shadow or bag , then your AI is stupid and buy some better sensors and improve your AI.

> our code is too stupid, because is not an actual intelligence

Well you don't ask for much

My takeaway from that article was that there's not enough trust in the raw inputs and possibly a missing continuous weighting system of the sensory interpretations. (humans do this naturally, weighing their various sensory interpretations in real-time simultaneously)

I can't find the specific comment now, but on previous autonomous driving discussions here, it's been asserted that Tesla (and many other systems) simply can't reliably detect stationary objects at speed. If the systems actually did a naive, "that looks like a stationary object, I'm going to avoid it" they'd barely be able to move in a highway setting.

That seems like an enormous caveat to the system overall. Do you know if there is any public justification for going along that route, omitting radar, other sensors, etc.?

AFAIK, it can detect them just fine, but it is bad at discriminating those its collision avoidance part should pay attention to from those it shouldn’t pay attention to (e.g. the road surface, traffic signs on the side of or over the road), so they (¿mostly?) chose to ignore all of them.

That just means it lacks the ability to place objects in 3D space and plot a course, which is... sad.

Most collision avoidance systems aren't that sophisticated. Radar based systems like those used in Volvo's Pilot Assist suffer from the same inability to discern stationary objects.

Most systems are self contained modules with low power micro-controllers that can emit basic warning and alert data onto the CAN BUS. Tesla's is actually taking the input from it's sensors and attempting to do more in-depth analysis and actually produce a 3D space.

The fact that they're still suffering from the same problem as other manufacturers shouldn't be an admonishment of their technology or attempts unless or until it's proven that their approach is fundamentally worse.

Well, no. No system that is unable to detect and map out a 3D maps of obstacles and plot a course through it without hitting stationary, well visible objects, is not suited to being used in a self-driving capacity. Whether it's from Tesla or anyone else.

If their approach fundamentally means they decided to work around mapping their surroundings in 3D correctly, as the GP suggested, there is a strong reason to admonish.

There are systems that can detect stationary objects, Tesla does not want to use those more reliable systems, so they use the programming methodology of tweaking tweaking the code when a bug appears and hope that was the last one.

Sure some AI gurus will claim that humans can detect this objects only with 2 eyes but they forget to mention that behind those 2 eyes is a brain an actual intelligence that is aware of the world and it's place in the world. So if you can't create an actual intelligence then at least put the effort in feeding correct data into your AI and remove the line of code "if static object ignore because too hard to detect if is a truck or a shadow/paper bag/sign"

> tweaking the code when a bug appears and hope that was the last one

Tesla should also be prohibited from pushing updates. When the software is updated, the code should be tested for at least X thousand miles and X months before a new version may be pushed. Letting an authority do the pushing avoids the temptation of manufacturers to push updates prematurely. It also solves the problem of unexpected changes for the user at random times.

If safety issues are detected, the automobile (or at least the autopilot) should be disabled until the code has been fixed and fully tested.

Yeah they want a low unit cost on the camera system, so they are trying to machine learn their way out of the problem. It's not going to work.

A self driving system needs to have a 3d understanding of it's environment. It's not acceptable to crash into paintings by Wile E. Coyote.

From what I understand there is also some code that is there to ignore static objects because maybe are signs or bridges and the AI is too stupid. So you have bad sensors with bad code combo.

Yeah it's essentially impossible for the system to be good- the limitations force other limitations.

> There are systems that can detect stationary objects, Tesla does not want to use those more reliable systems

What systems are you referring to? Lidar and radar systems can't reliably detect stationary objects as hazards, because most of the world is stationary and consequently gets filtered out.

Filtering out is the problem that develoeprs/engineers should solve right? I can't accept the idea that the car can hit a wall because "our devs are too incompetent or we are to cheap to use better hardware" , a wall or a truck does not look like a bridge or a shadow or a sign, if the AI is too stupid then is not ready, even if is better then a drunk teen that drives a 15 years old car in some very specific situations.

The lidar/radar detects the objects the AI should label them and decide, if it can't do it then throw more video cards at it and more training or find some competent people that could use more deterministic code to reduce the complexity before the AI attempts to run.

Yes, in principle there are systems that could detect stationary hazards. But as far as I'm aware the best lidar and radar approaches are currently no better than Tesla's RGB approach. It seems that the AI is much more important than the sensor.

No Waymo car has crashed into an emergency vehicle yet.

Waymo has driven 20 million miles. Autopilot has driven (about) 4 billion miles.

The sensors could detect a solid object, the issue seems to be that the sensors will detect many things and the AI or the hardware is incapable to handle the amount of sensors data.

Though the Google car seems to have strong enough hardware to keep track of many vehicles and objects at a big enough distance to not be surprised if something appears or disappears for a frame.

Not sure what people want for a sensor, to detect and label the objects for you? or calculate velocities and predict the future? That seems to be the software developers job but you might need competent people and not throwing big data into a black box and retrain until it mostly works.

The problem is with the sensors, not the AI, and occurs across the industry and isn't exclusive to Tesla. Volvo's Pilot Assist has the same short comings.

https://www.wired.com/story/tesla-autopilot-why-crash-radar/

From my reading of your link you are wrong.

How a code that decides what to ignore or not is a sensor issue? Sensors tell you that there is something in front, sure maybe is a sudden thing and there is no velocity/acceleration vector for a few moments but the object is detected ...the software DECIDES what to ignore.

The example where a car changes slane and the code ignores any static object in front of that car seems a clear case of a crash waiting to happen, the car in front might change the lane to avoid something. (and the fact that some other tech that is honestly labeled lane assist has this issue does not excuse a self claimed autopilot to have same flaws."

Let me summarize and let me know where I am wrong:

"Because the software is bad and it triggers too many false positives the coders added a "clever" hack , we ignore most static objects."

> How a code that decides what to ignore or not is a sensor issue?

I guess "Sensor Module" would be a more accurate description.

Most automotive systems are designed as a series of modules that function independently but jointly form a system. They either broadcast or consume messages over the CAN BUS.

The Volvo V60 for example has a Forward Sensor Module, the radar, which outputs a handful parameters like Primary Target Range, Voltage, and Automatic Alignment Offset. It's the responsibility of other body modules to consume that data. The Closing Velocity Module might then broadcast Acceleration related messages which intern would be picked up by the Adaptive Cruise Control Module. Or it might broadcast Collision Warning Messages which the Stereo Module reads and responds by blaring a warning noise out the speakers and also the Brake Control Module receives and applies the brakes.

If that's the case, the correct solution is to shut Autopilot and all systems like it down, and charge their developers with reckless endangerment.

A autonomous driving system which can't detect stationary obstacles is not an autonomous driving system: It's a murder machine.

If they can't do this, they shouldn't operate, full stop.

The horrible truth is that if autonomous driving can save 1+ life, it's better than what we have now so we should keep it and accept that crashes will occur much like they do now with humans in sole control.

There's no reason to kill autonomous driving just because gen 1 - 50 isn't perfect but is an improvement on current driving methodology.

Giving a one-line sentence of what happened disregards other variables such as, lighting conditions, where the stationary object was (Over a hill, round a sharp bend, etc.), how much stopping distance was there. A lot of Tesla hate is purposefully done from EV haters & Musk haters. No one can argue that there isn't an establishment who wants to tear down and prevent Tesla & Musk from succeeding. Teslas will catch on fire (like other cars do), Teslas will crash (like other cars do), Auto-pilot will fail (while also being overall an improvement on driving safety).

We can't assume the stationary object was 300 feet away on a straightaway in a perfect visibility conditions, we have to judge whether a human would've performed worse and whether the same cases are occurring over and over. 12 crashes over a few months is a pretty tiny fraction when there are probably more than 12 humans crashing into stationary objects every hour.

I do not agree with this argument

>> The horrible truth is that if autonomous driving can save 1+ life, it's better than what we have now so we should keep it and accept that crashes will occur >>

As that '1 life' saved may be countered by a number of lives lost due to the system that might otherwise have survived.

“The horrible truth is that if autonomous driving can save 1+ life, it's better than what we have now”

This is not true at all and we have much higher standards than this. This is like saying if a vaccine cures Covid but manages to kill people at a rate slightly less than Covid, we should embrace it.

Isn’t that exactly what we are doing? (-slightly, as we would need to define the threshold)

>can save 1+ life, it's better than what we have now so we should keep it and accept

So I take it you would be fine with the nightmare version of Apple's phone scanner, continuously scanning your devices for thoughtcrime? After all, "If it saves just one child..."

Any time I hear emotional appeals that ignore the cost side of the equation, I look for a financial motive.

Let's rephrase. If the important upsides and downsides are all represented in the number of traffic fatalities, and that number goes down, then that's good.

The problem with the Apple system is significant downsides that are completely separate from CSAM.

>if autonomous driving can save 1+ life, it's better than what we have now so we should keep it and accept that crashes will occur

This argument ignores the reality of driving: by being a competent, attentive, careful driver you can avoid many classes of accident. Crashes are all about fault, and as a safe driver you can avoid most at-fault accidents. You get to avoid all the self-inflicted damage of drunk/incapacitated driving, unsafe driving for weather conditions, extreme speeding / reckless driving, etc. You still have to deal with the externalities of those, but some fraction of the time you can notice and anticipate these people on the road and stay away. Not always of course, so that still leaves some number of deaths per mile, but I'd bet that threshold is way lower than the "save one life" bar. So if you're a good driver, autonomous cars at the "current death rate" safety bar would make you less safe.

What you get by ignoring this and using the current death rate as a bar is a "transfer of deaths": you take all the deaths that were the fault of the driver (driver dies due to being drunk/high, very reckless driving, etc) and transfer them to random innocent people who were not at fault.

> This argument ignores the reality of driving: by being a competent, attentive, careful driver you can avoid many classes of accident.

It’s funny you use the word reality since this is completely at odds with driving in America. You’re comparing an idealized world of competent drivers to the real world where there are basically no barriers to getting a license and everyone is on their phone behind the wheel. Unless you advocate for stricter licensing and more police enforcement against inattentive driving with equal vigor, this line of reasoning doesn’t hold.

The systems you're referring to are radar based systems. Tesla stopped putting radar in the Model 3/Y in favor of it's vision system.

It would be interesting to see a breakdown of the 12 incidents and the Tesla Models involved. Were the incidents involving radar/vision blended, or vision only vehicles or both?

If the incidents only involved older vehicles that were radar equipped, that would be good PR for the vision only system. If the incidents were in vision only systems, it would be bad. I suspect it'll probably be a little of both.

Does this happen more often per mile driven than human drivers?

> It doesn't really matter. They're stationary objects; there's absolutely no reason, good or bad, for the algorithm to determine that the correct trajectory change is into the stationary object.

Are you saying it doesn't matter how often it happens? I'd be a lot more worried about things if something happens always compared to it happening in some cases. Sure, stationary objects should be driven into, I agree. But if this only happens 1 time over the course of 10 years, I'd be a lot more lenient than if it happens once per week, no matter if the reason is "good or bad" (not sure what that means?)

There is enough information here though.

These are situations where failure is not suppose to happen when the driver is paying attention. Given that the auto-pilot is always paying attention when operational, this becomes an unacceptable failure mode because it failed in the most rudimentary way - failure to perceive stationary obstacles and a display of complete lack of situational awareness.

For emergency responders on highways, there are means to reduce the risk of additional collisions by increasing visibility to oncoming drivers (markers, lights, etc.). If the auto-pilot is not able to perceive these even when paying attention, we do have a problem.

It’s not that simple. 12 out of ten million is extremely bad if the average driver takes part in that lottery 100,000 times a day. 12 out of 15 is nothing to worry about if the event is extremely rare.

I guess this lies somewhere between these extremes, but wouldn’t know.

We also don’t know how many of these accidents were prevented by attentive drivers.

When human drivers cause accidents, the causes are collected and analyzed, and laws are proposed to prevent future accidents from happening. It doesn't matter if it's 12 or 12 thousand, or even zero crashes... if there's fear, then society will demand something be done.

The fact that AI can give no easily understood excuses, and improvements to the algorithms is a trade secret means that any fear cannot be abated, aside from outright banning of AI driving.

Thanks for your honesty.

Hopefully the NHTSA is also going to investigate the many other cars with traffic aware cruise control, which have the same problem with stationary vehicles.

I was driving my Rav4 with TACC enabled, and it would have crashed into a column of stopped cars if I hadn't noticed in time and intervened.

Tesla has something like 1000x greater rate of hitting stationary emergency vehicles. The fact that all the other cars have the same issue with tacc actually points even more strongly to something being wrong with Tesla.

> 1000x greater rate

What's your denominator? Is this per year or per kilometre driven with TACC active?

citation, please.

The driver is responsible for stopping the car. You literally agree to it when you enable AP. People are too comfortable and not paying attention.

The fact that it occurs so often should suggest to you that autopilot itself makes people too comfortable and not pay attention.

I, and many others, take the position that there really isn't any practical gradation between more-or-less conventional driving, maybe with light lane switching assistance or something, and full on "Level 5" autonomous driving, because the idea that human brains can watch something for hours at a time only to take instant, sub-second actions every few hours is an insane model of human cognition.

I don't disagree with the idea of having a gradient to measure our progress up to full autonomy, but I haven't seen much to say that we should deploy the intermediate stages. Until autonomy can be reasonably guaranteed at at least human levels of safety under the assumption of no human control, it really shouldn't be deployed. "But the human had 673 milliseconds to take control and avert the accident! It's their fault!" is simply not a reasonable defense.

Shouldn't the objection be to anything less than Level 4, rather than Level 5? Level 4 requires that the vehicle can safely stop itself and wait for a human to take over when it is faced with a situation it can't handle.

(There's an argument that if you take away the routine parts of driving, humans won't be able to deal with anomalous situations, but that's a different argument than the attentiveness one.)

If that is possible, yes. I'm not convinced it is possible to create an autonomous vehicle that only panics in situations where there is a safe pull-over, or, if such a vehicle exists, that it'll be considered "autonomous" when it can't go past 25 or 35 mph or something. There's plenty of driving situations that may be confusing where pulling over and stopping isn't possible.

(And, to be clear, I'm not asking for perfect safety, I'm asking for "human or better", and also understand that there may be some interesting fuzziness near the border where the two have different strengths and weakenesses but come out statistically the same. However, I don't feel like we're there yet.)

Mixed control between automated and human for tasks that require persistent attention is bad. We have known this for decades. The example often given in engineering textbooks is 3 mile island. I suspect that the engineers involved in self driving systems know this too, but the liability is too great to make the switch all at once. It seems better to take baby steps into self driving, but history teaches us it should be all or nothing.

You could say the same about airplane autopilots, but they have an amazing safety record.

Airplanes also have 2 pilots and nothing to crash into.

Far fewer objects to hit in the air: disengagement means "keep going this way," rarely "and crash into an eventual obstacle". Plus, their early safety record is written in blood like everything else is.

> Plus, their early safety record is written in blood like everything else is.

Yeah, and that's how it's going to be with car autopilots. Which is why it pisses me off that people want to stop progress on it.

Interesting typoes in "be unwilling test subjects". Feel free to volunteer, though. Elaine Herzberg sure didn't.

The airline autopilot doesn't have to negotiate anything like the kind of hazards that a car driving system does. Also, pilot training, the presence of a whole flight crew to monitor and catch errors make for a completely different situation.

Not to mention you have to be licensed and spend hundreds of hours to share controls of a commercial plane.

Car AP is way harder than plane AP and saying "But planes can do it, why is Tesla so bad at this" is facetious.

They have their own records of fuck ups [0]. And for similar reasons as the Tesla AP crashes. Autopilot disengages and leaves the pilot in the lurch. If the pilot doesn't have situational awareness right away it can be fatal.

Airplane autopilots, depending on generation, are full touch. The pilot programs the waypoints. The pilot is alerted and agrees to every course correction and change.

[0] https://en.wikipedia.org/wiki/Air_France_Flight_447

Airplane autopilots don't disengage. They do what you told them to do. They are simple and predictable systems that at most simply follow a predetermined course exactly.

That's not entirely true, there are situations where they do disengage automatically (e.g. due to missing/conflicting sensor information). By the nature of the thing it is a) rare and b) the pilots have more time to react because it won't be a "you're going to hit a truck in 1 second" situation unless something went badly wrong before. But there is still some risk to the handover situation making it more likely pilots misinterpret the current state of the plane or do not realize which automatic support is still active (not just autopilot, also other avionics support (e.g. if I remember correctly one aspect around the Air France 447 crash is that the pilot likely did not realize that the plane had disengaged support systems that would keep him inside a safe flight envelope in normal circumstances).

> The driver is responsible for stopping the car. You literally agree to it when you enable AP. People are too comfortable and not paying attention.

Autopilot encourages people to not pay as much attention while driving, until a critical situation comes up at which point they're expected to be fully in control. That's a design flaw that isn't fixed with a liability shifting "Agree" button.

It will nag you every 17 seconds (if not attentive/holding the wheel with a torque) and abort auto pilot completely if you do not comply.

If you do this three times in a drive you have to power off and on the car again to be able to use autopilot again.

I've done maybe 40.000 km on autopilot and it's very aggressive compared to my other car (ioniq electric) when it comes to attention.

The fact that some people purposely defeat this system with weights on the wheel is another matter.

But saying that tesla encourages you to not paying attention is not correct.

To be fair, the person you are responding to may not like those other systems either.

All the articles with industry analysts insisting that other driver monitoring systems are nearly perfect have sometimes blinded me to that fact.

A lot of people don't like any of these systems for a lot of the same reasons that people didn't like basic cruise control in the early days.

I haven't used basic cruise control in years. It's actually convenient not to have to pay attention to the speed but I find, on roads with even moderate traffic, it comes at the cost of prioritizing maintaining a constant speed rather than flowing with traffic and generally staying in a lane.

You have no telemetry and no idea what actually happened, car might have veered without enough warning. Which is the whole point of having an investigation, really.

Reply on news.ycombinator.com