Hacker News Reader

This is the third known FATAL US autopilot crash. There are other non-fatal crashes. Many with video.

Fatal crash in China: [1] Rammed into street sweeper at full highway speed.

Non-fatal crash in California.[2] Rammed into fire truck stopped on freeway.

Tesla's system does not detect big solid objects in front of the vehicle. It's "recognize rear of car", then "brake for car". No recognition, no braking. Machine learning at its finest.

I know, I've posted this before, but I keep seeing denial.

[1] https://www.youtube.com/watch?v=fc0yYJ8-Dyo

[2] https://abcnews.go.com/Technology/utah-driver-slammed-tesla-...

My experience with machine learning and vision systems amounts to one lab assignment a few years back, so forgive me if this is a stupid question - but why is there not a 'complementary' hard coded system built in that can override cases like this? I can think of no reason as to why stopping a car driving full speed into an inanimate object would be a thing that a human can't code. Surely there must be something in place - an "if(hurtlingTowardsWall){ignoreAI; stopCarQuick;}".

How do you know there's a stopped object in front of you?

Don't the Tesla have forward facing radar?

Traditional radar is bad at spotting stationary or slow moving objects, and to avoid false positives at every traffic sign or other stationary object above or next to the road, any radar-based object detection has to be tuned conservatively or the car would be flipping out all the time.

That’s why other companies use lidar, which is much better at object detection, but has its own problems (like fog or heavy rain, for example)

Stationary objects are very detectable with radar as they have relative motion to a moving vehicle. Also objects with no relative motion are only tricky for CW doppler radar.

The issue is mostly your last point. The radar sensors used have low resolution and would have trouble distinguishing a turning semi from an overhead sign.

Parallax via optical cameras. Like the human eye or Subaru's Eyesight.

I use Waze during my daily commute and it regularly gives me a warning about a stopped vehicle on the road or side of the road; would a crowdsourced system like that help for autopilot so they change their algorithms slightly to detect stopped vehicles more better? Or just reduce speed? Or alert the user and disable autopilot well before the location?

I'm aware that would be creating an exception for an exceptional situation and not really a root cause solution, but still.

I'd be extremely extremely hesitant to throw on more dependencies for life-death scenario handling.

"Unfortunately my son didn't pay his Verizon bill on time, so they cut him off and since his car couldn't get updates it rammed him into a fire truck".

Apparently many or all of the current self driving cars don't do well when running into fixed (ie stopped) physical objects. It must mean somehow in the way that they detect things, they must have to discount stopped objects, and 'moving objects', like cars are detected in a different way. I'd love to hear more background about why this is the way it is.

I agree that when you can't handle this kind of situation in a tesla it really challenges the idea that we'll have total self driving in a year. There must be a lot of software that is under testing that could be more reliable than what they have today, yet the issue is still there. The last part (5 or 10%) of regular software dev is really hard, and it should be even harder for self-driving, never yet approved or safe enough self driving software.

I predict we won't have fully autonomous ("total") self-driving in the next 25 years. To think it will be out in the next year or even in the reasonably expected lifespan of currently shipping Teslas is utter folly, IMO.

When I see long haul trucks driving along highways with humans only taking over in the populated areas I'll believe full self driving may only be a decade away. Tackling the full self driving consumer market when there's lower hanging fruit makes no sense.

They could wipe out Über.

Based on what?

It seems this is a classic estimation problem, and we need performance data points to fit a curve. Do we have anyting? Do we even know if this is a problem where we have a chance at accelerated compounding (so exponential improvement rate)?

Based on the fact that we don't have acceptable performance on clear, sunny, well-marked roads today, I don't expect that we're within a year of handling snow-covered roads, road detours, human flagged construction details, etc.

I think the self-driving car efforts are dramatically underestimating the effort required to close those gaps to zero at least in the hype machine but also perhaps in the daily working cycle. (That's all well and good; I'm sure that any audacious engineering program should discount some of the hardest parts lest everyone get discouraged and never begin.)

As far as I remember, it is because radar gives you too many points that are usually false positives if you don't discard stopped objects (which include signs, guiderails, anything on the border of the road (which you'll be driving right at in a corner, after all)). This is something that ACC in all radar-using cars does, AFAIK. AEB works differently in that stopped objects are tracked, but it only works well at slower speeds. If I had to guess it's because they have to track a potential stationary object over time to determine that, yes, it's an object that concerns us, it's in the way, and it's not just noise.

In case of the first trailer crash it apparently looked similar to an overhead sign to the sensors, also approaching a stopped object would be AEB territory, not ACC, which doesn't work well at higher speeds.

The first trailer crash wasn't just that it looked like a sign to the vision system. It's that the radar saw it and alerted, but the AEB was configured to do nothing unless both the radar and the vision system alerted. Since it was a white trailer against a bright sky, the vision system couldn't see it.

Tesla later pushed a software fix for the AEB to trigger even if only the radar alerted.

https://electrek.co/2016/07/01/understanding-fatal-tesla-acc...

Step 1: identify where stopped objects are relative to the sensor

Step 2: if stopped object is immediately not in front of the car, otherwise, ANALYZE IMMEDIATELY because a stopped object is right in front of the car.

It's not rocket science, and it's something so ridiculously basic that Tesla's failure to do something this simple borders of gross criminal negligence. It's something their competitors already do; in fact Waymo's problem is that it's too sensitive to stopped objects in front of the vehicle, hence the Waymo's constantly stopping and starting or driving very slowly...but that's the better and safer way to do it, especially when you're testing on real streets with third-party vehicles.

It's an issue that needs to be fixed but it's not extremely simple.

The car always has still-standing objects in front of the car (the road, anything around the road, barriers, anything on opposite sides of barriers, things in the direction of the ar but the road is turning etc.

The car has to know that the road doesn't slope up over the truck or around it. I'm guessing that's where the split-barrier fails occur.

It really needs to create a real-time 3d map of the entire environment (including what's visible from eye-height, not only front of hood) and map a route through that.

The car always has still-standing objects in front of the car (the road, anything around the road, barriers, anything on opposite sides of barriers, things in the direction of the ar but the road is turning etc.

Right, none of which are directly in front of the car when the objects and care are plotted in 3d space. Easy (but not trivial) to do with LIDAR but not yet doable with purely visual systems like "Autopilot." Which is my point, and which is the reason every other self-driving car company uses a combination of sensors to get data about car's surroundings.

Autopilot is not self driving. The human is still driving the vehicle and the human is still responsible for stopping the car from hitting things.

Well, then what is it doing? Crashing your vehicle into semis when your attention lapses for a second?

Then it should be properly marketed as autocrasher, not autopilot.

An autopilot that requires that you have your hands on wheels and pay attention at all times is missing the auto part of the autopilot (reminder that auto means self). So such a system can't be called an autopilot.

It's not too bad if you compare it with the meaning of the original word as used in aviation context: a relatively primitive device that automatically keeps the trajectory of the aircraft (regardless of what is ahead by that trajectory). But it does sound misleading when presented to the general public (as opposed to highly trained professionals, which is the case in aviation context).

It’s auto pilot where pilot means the person steering ng a ship. There is still executive oversight required as in a captain or at least a navigator. It’s not “autodrive”.

sigh Piloting here means "controlling a vehicle", i.e. driving. As in "Formula 1 pilot". Do you think "Formula 1 pilots" are sailors?

It's analogous to an autopilot of an airplane. It keeps going straight, direction, speed, altitude. An airplane can fly straight and level right into an airplane. For some reason (partly Teslas'), people look at a tesla autopilot and think it does everything reliable. They used this misleading name, but people should understand it same way they understand a plane's autopilot.

Tesla calling their driving assistance "autopilot" is a marketing move. I don't see why we should accept their definition. Autopilot to me means that the human pilot can remove attention from immediate steering tasks. This is not true for the Tesla "autopilot".

AP is like a student driver. Perfectly safe as long as the instructor is there to take over with very short notice. So saying AP is safe because % crashes in $ millions of miles is pretty irrelevant. AP only drives with supervision and when this is missing it crashes.

How many times did the driver have to intervene to avoid the crash? Tesla provides no data about this. But next time you hear “AP is so safe it only crashed 3 times” just remember that student drivers also almost never crash. As long as it needs constant attention, not considering this in the statistics makes them irrelevant.

To my understanding, this is why they don't have a real model of the world. They don't understand that the road signal right ahead of you poses no danger, because you're actually in the middle of a curve and your trajectory won't actually intersect it. Or that driving below a bridge is fine, even though you're pointing straight at it.

Once they are sophisticated enough to have an actual world model (something which, to my knowledge, only Waymo actually has ATM) they can stop ignoring static objects. Until then, they would keep slamming the brakes every 30 seconds for all the stuff they don't understand.

They probably bet on having the DNN figure this out from the data.

Of course this seems like a really hard thing. Probably with enough layers and with the right learning tasks it might be able to do it eventually. But the probability for noise and various learning artifacts is enormous, when you want to learn a sort of model of everything road related.

> they must have to discount stopped objects

Well, there are just a lot of them, outside the lane markers, and those never kill you. And while the systems surely try to classify parked cars as well, detection quality isn't under much scrutiny. Then when that one exception is sitting in the lane, good luck, your life depends on a second class feature.

Anecdotal evidence suggests that a similar perception vulnerability exists in long distance cyclists who spend many hours on the own road: lanekeeping and pothole/junk avoidance get more and more delegated to subconscious processes which work with eerie (and enticing) reliability. But that "mental autopilot", tuned to the open road, knows the back of cars only as something that pulls away from you, the simple model knows them as something not worth "raising an interrupt" over. Similar learned routines certainly exists with highway driving as well, but when driving, there isn't a class of objects which is routinely there but almost universally not a concern, like backs of cars are for a cyclist. Reports of that cyclist hitting a car parked in the open road always earn a lot of ridicule, but it's not quite as surprising as it seems at first glance (the very majority certainly succeeds at "disengaging" in time, but the exception make the news, same thing in every field)

> Teslas with Autopilot on crash roughly once every 3 million miles.

I also wonder if this statistic includes situations where the autopilot had disengaged immediately beforehand and the human had failed to respond.

If they named 'auto pilot' 'auto driver' it would at least be possible to have a sane discussion about the extent of the gap between the name and the actual performance. Since they named it 'auto pilot' we can not have that discussion because 'auto pilot' is a term that applies to aircraft, not to vehicles and you can not expect your average driver to know what the auto pilot in an aircraft can and can not do. Presumably laypeople will assume that an auto pilot in a plane is more complex than one in a car because they perceive aircraft as more difficult to operate than vehicles (and rightly so).

> Pretty much all self-driving companies have disengagement rates way worse than humans.

What’s the human disengagement rate?

When you fall asleep at the wheel? No idea…

Exactly. The parent casually asserts a difference in a measurement that doesn’t apply to humans. We can’t usually disengage to a safety driver. The comparison is not meaningful.

Parent comment compares human crash rate (1/300,000 miles) to self-driving disengage rate (1/10,000 miles). With the assumption that the self-driving system would have crashed had the safety driver not intervened (a reasonably safe assumption, given the purpose of having a safety driver in the first place), this seems like a meaningful apples-to-apples comparison to me.

I don't think that the general human crash rate is appropriate to compare. It makes more sense to compare, say, a professional bus driver.

As it happens, my uncle drove for a living and I recall he had a plaque for a million miles without an accident.

Or "Jesus, take the wheel!"

Perhaps because the first step to achieving anything is to believe firmly in it.