Tesla owner almost crashes on video trying to recreate fatal Autopilot accident | Hacker News Reader

Tesla owner almost crashes on video trying to recreate fatal Autopilot accident

electrek.co

74 pointsevo_957 comments

Previous discussion from a few hours ago: https://news.ycombinator.com/item?id=16732436

Also worth noting that this video is from a highway in Illinois, not California.

Note that I'm not saying Tesla Autopilot is ready to bet lives on, etc, etc etc... but I think we also need to address the problem of bad lane lines for humans too. The right side of the divider was virtually invisible. Anyone fully paying attention should also see the right-side lane markers continuing straight and stay with them, and see the barrier very early (although I usually see those with a lot more reflective markings and barrels to absorb energy), but - bad lane line markings can definitely throw off imperfect humans plenty too. The I-15 in Utah is notorious for this. Areas that have been under constant construction for years have layers of previous temporary lane markings that are badly removed and weren't clear to begin with, and you can be driving along with several lanes of traffic who suddenly get confused about which lane they should be following and start converging into the same lanes.

Clear lane markings are not a requirement just for autopilot.

The unclear-ness of the lines stood out to me, as well.

Without a sensor-based roadway, I'm not sure how systems like this can be possible. For instance, very-wet roads at night have so much glare from headlights it's impossible to someones see the markings on the road, especially without any reflective/haptic coatings.

The lack of lane markings and missing crash barrier are symptoms of deferred maintenance. Why would a sensor based roadway be immune to similar neglect?

In a best case scenario, I'd think that sensors would have the advantage of perfect clarity on when they are working, and when they aren't. Eg, if a stretch of sensors goes bad, cars would encounter them and immediately attempt to safely stop, pull over, etc.

With this, we have AI having to predict when it does and when it doesn't know where the road is. A sensor approach could be akin to a railway, and the rails are either detected or not.

A merge of the two methods would still be needed of course, to detect obstructions in the roadway, navigate safely in the emergency event that no sensors are available, etc.

It would also be cool because we could define when and where those new roads are available. You wouldn't be able to drive AI to the Oregon Coast and have it suddenly fail on an untested cliffside. The road is either AI friendly or not.

I'm sure there's flaws in this line of thought, I'm merely speculating.

Semi related, but in rainy weather (Washington), there's plenty of times where I'm basically guessing at the line location. I'm not sure if I'm especially bad at it, but with light reflections all over the place, I lose track of those little white lines.

This gets made worse then construction strips old lines and draws new ones a couple feet over. The old ones in dry weather are barely visible, but in watery weather, they hold water differently than normal road surfaces.. meaning there are now lines which, due to different water levels of normal road surfaces, form visible lane lines competing with the real lane lines.

The reflective bumps are a huge boon for me when they exist. I'm always surprised more accidents aren't caused by poorly visible lines.

I agree that lines should be fixed, however humans rely on things other than lane lines, and autopilots should too. When you can't see you get cues from other cars, estimated distance from both sides of the road, sign placement above your head or on the side of the road, etc...

If we need to repaint all roads to make them work for auto-driving cars, we might as well just move to one of those simple systems like where a robot follows a pre-drawn line on a piece of paper. Maybe throw some bar codes in there so the car knows where it is.

If the cars can't deal with environments that even humans can, I'm not sure what the point is anymore. I expect a self-driving car can see through things like fog and darkness better than a human.

Humans primarily follow lane markings. When these are missing and/or unclear it becomes very difficult to guess where you are supposed to be.

Anyone who has driven through road construction and had to follow temporary lines while ignoring blacked out lines knows this struggle.

>> If we need to repaint all roads to make them work for auto-driving cars

No I'm saying the section in the video is in need of repainting even for humans.

And I did not disagree, however if auto driving cars can’t deal with that situation at least as good a humans, then we have a problem. Obviously we’re at 1.0, so things will get better.

Feel like the system should be made of layers. A rational one that follows lanes and rules and such but also a more reptilian backup system that self-preserves. The latter can't be used all the time because of danger to others but it's similar to the evasive moves someone would take in danger.

This same pattern of trying to make a perfect system instead of multiple layers by acknowledging lack of perfection exists quite a bit in tech.

101 southbound has a carpool ramp to 85 that solves this problem by using an exit only lane, which works remarkably well at keeping cell phone drivers from ramming the median head on. Tesla MV drivers, how does autopilot fare there?

This is pretty shocking, more so than any other autopilot related news or videos I've seen. It's something that no normal driver paying attention would ever do.

This further confirms for me the feeling that autopilot in cars won't become even remotely common for another decade, especially in any parts of the country with poor weather.

There will be bugs.

This is the beginning and I only find it interesting and not in any ways shocking.

From my understanding the driver who died in the crash had previously reported problems in this area. I wonder if it was this specific crash site or not, I wonder why too if the person was aware of it why they didn't pay more attention during it - and to stop trusting it. I can understand how easy it is to be uncertain of something and have enough trust or rather a thread of hope that "nothing bad will happen to me."

I wonder too, if it's true that the person who died in the crash reported the area, what process Tesla followed and .. Any area reported with a problem IMHO should immediately start warning other drivers going in the same area until the issue is resolved, even disabling auto-pilot completely until Tesla's resolved it. I'm sure that would make people safer than the temporary inconveniences they would cause - I'd hope people would take that attitude anyway.

If the driver previously reported problems on a section of roadway and was then inattentive when driving that section of roadway, that’s a perfect setup for a reckless driving ticket. They knew a problem existed and recklessly ignored it. Autopilot does not excuse this in any way whatsoever.

Probably best not to think of Tesla as the standard bearer.

You can go back through Animats comments here and see criticism of the (apparent) approach Tesla is using, for months and months:

https://hn.algolia.com/?query=by:Animats%20Tesla%20map&sort=...

And yet, someone had destroyed the crash barrier prior to this fatality. Locals report this happens often.

An autopilot does not have to outperform a human driver in every imaginable situation as long as we make smart tradeoffs regarding what the machine can and cannot deal with and there are still significant advantages over humans in the managable common case.

There may not be a solution in sight for making robots deal with badly marked roads for example, but we might be able to just sidestep the problem with better road markings or more conservative handover-to-human protocols in areas with improperly marked roads.

> Then it seems like Autopilot’s Autosteer stayed locked on the left line even though it became the right line of the ramp. The system most likely got confused because the line was more clearly marked than the actual left line of the lane.

This doesn’t surprise me at all. I test drove an “autopilot 2” car once on a local road at 30-ish MPH. It appeared to try to drive centered between the lane line and the curb, totally ignoring the fact that the right half of the “lane” was parking. It then tried to drive me into a parked car and later into a trash can. Unless they’ve improved dramatically, this is the same thing.

Autopilot seems to be a mediocre adaptive cruise with mediocre steering assistance that pretends it can actually drive.

Autopilot should only be used in free ways...

Then the software shouldn’t allow it to be enabled off of freeways. And it should be able to dodge big concrete and metal dividers.

Then read the user manual and use only accordingly.

Tesls have made out that the driver ignored the warning; but it didn't actually say that. It said previously in the drive he ignored that warning, but phrased it cleverly that makes it sounds like during the accident he ignored the warning.

Feel free to correct me if I'm wrong; but I'm 99% sure that Tesla managed some pretty spectacular weasel wording there.

Yes, their statement included definitely a lot of room for interpretation. I was surprised that most media turned it into a story that took Tesla at least a little out of the line of fire. Especially here in Germany, were media (and most car enthusiasts) generally wants to see Tesla going bankruptcy asap.

> Especially here in Germany, were media (and most car enthusiasts) generally wants to see Tesla going bankruptcy asap.

Why?

Actually there are multiple reasons.

First of all a lot of people here believe that our cars are the best in the world and no one could even compete close. A lot of this is based on tradition, think about that gas and diesel engines were both invented by germans. A foreign company getting so much hype just doesn't fit to their ideology. Especially an american one, there is no general bad reputation for american products here (rather the opposite), except for cars though. American cars are considered generally as poor made and technological far behind here. A Tesla (who is this anyway?) more innovative than a Mercedes, BMW, Audi or even Porsche??? It cannot be, what must not be.

Then there is still a general rejection of electric cars by most people, it's ridiculous how many fake news gets widespread around that topic here in media, always to worship the diesel engine as the holy grail.

The only reasonable point ist that there is a big fear that electric cars will cause a lot of people to lose their jobs, because they are much easier to build. Keep in mind, that the automotive industry is the heart of germanys economy. But instead of accepting that technologies change over time and supporting and pushing our industry to make a good transition to keep their world wide market share, they still hope this could be prevented at all in some kind.

Man, I panicked just watching that. I can only imagine how I'd react if an "autopilot" started beeping, forcing me to take over in an already bad situation.

I'd love to own a car with all these sensors, but currently all I'd want to use them for is keeping distance to cars in front of me, and stopping if needed/etc - ie, advanced cruise control.

I'm terrified to drive a car that could decide to fully apply the brakes if it deems necessary. Imagine if it falsely detects a human in the middle of the highway when you have a car following closely. Very unlikely scenario, some would say impossible... but I'd rather just pay attention to the road myself instead of getting my car to do it for me.

There have been multiple times where, despite my best efforts to pay attention, I couldn't even see a traffic altering incident in front of me. The car in front of me, which was the only one I could see, did not prepare me quickly enough. Now, I didn't crash, but an advanced warning system can LIDAR/whatever under cars to see many cars ahead, warning me if three cars ahead are traveling at 0mph and I'm traveling at 70mph.

Of course, safe driving distance is important there - I'm not claiming otherwise. It was my safe distance that made it possible for me to stop. This is entirely besides the point though, as my human senses were entirely letting me down in that situation.

There's actually a good video of almost that exact situation by.. I think it was a Tesla. It started beeping with little visible reason why. Turns out, the traffic two cars ahead was at a stand still.

I can see what you're saying about automation, but I still think it would be a net gain in most scenarios. Though, I imagine it would be best to let people turn that off, but keep warnings intact.

Though eventually that hypothetical car behind you will also have autonomous breaking, so it'd be fine. It's just a matter of getting adoption rates high enough.

Then the crash would be the fault of the car following closely.

If you slam the brakes in front of a car for no reason and they rear end you, you are most definitely at fault.

At least in my state, you're supposed to allow something like a minimum of 4 seconds or some length of space between you and the car you're behind. The argument is that the car rear ending you should have been far enough behind to stop.

Effectively, if you get into an accident with a car infront or to the side, you aren't at fault when the guy right behind you hits you too.

Unrelated to the autopilot stuff - this junction seems like an absolute menace to me. The lines are confusing and that crash barrier seems purpose designed to kill people. I've only driven a bit on US roads but I don't remember seeing junctions as bad as this. Is this common or an outlier?

I've seen off-ramps like this all over the US.

I've never felt unsure of being able to utilize them; but I've also never had a computer drive for me, either.

Well, I can tell you that as someone who has only driven in America that this road looks fine. Actually, it looks relatively good compared to a lot of our roads. Drive around some parts of Pennsylvania... blech. Road markings are the least worrying problem, giant potholes that will take your tire out are a much more pressing issue.

I had to watch the video twice because I thought the driver had messed up and dropped the camera before the important bit. Only on my second viewing did I realise that the white line was no longer the divider for the lane.

You can Just make out the white lines of the lane going to the right and chevrons between them and the offramp but I agree that, forgetting all the autopilot stuff. I could easily see a tired driver going straight into that barrier or even a non-tired one in bad visibilty taking cues from the white lines.

FWIW, this article is worded as if it's the same exit. It's not; the video attached is in Chicago.

Ramps like this are ubiquitous across the US, a self-driving computer unable to parse faded lane markings is alarming, to say the least.

If this is a reproducible 'bug' - then Tesla's stance of Autopilot is ... certified by US Govt to reduce accidents by 40% ... . "That does not mean that it perfectly prevents all accidents — such a standard would be impossible — it simply makes them less likely to occur." is going to severely discourage a lot of people from not using Autopilot.

Tesla's expectation that a driver keep their hands on the steering wheel within 6 seconds or they can die - is not going to bode well.

Also, "Internal data confirms that recent updates to Autopilot have improved system reliability." [2] is going to come under a lot of questions. What standards/tests do Autopilots even go under ?

[2] https://www.tesla.com/blog/update-last-week%E2%80%99s-accide... [1] https://www.tesla.com/blog/what-we-know-about-last-weeks-acc...

The types of collisions will shift to different concentrations as well, so there could be a "500%" increase in one type of collision, while a "20,000%" decrease in another area.

> "certified by US Govt to reduce accidents by 40%"

This finding, by the NHTSA in investigating the 2016 Tesla fatality, is under contention. A research group has filed a lawsuit under FOIA since the NHTSA hasn't agreed to release the data: http://www.safetyresearch.net/blog/articles/quality-control-...

Shouldn't autopilot at least stop the car because of the barrier in front of it? Even if it can't figure out the lane configuration?

Propably not. A car stopping because "its confused" in the middle of a high speed lane is dangerous. Slowing down OTH (as it is shown in the video).

Stationary objects are less likely to be recognized than moving ones: https://www.wired.com/story/tesla-autopilot-why-crash-radar/

I wonder why these systems aren't linked to GPS and maps, surely you'd be able to calculate that going right into the middle of two fork options isn't going to end well.

I've heard that GPS is only accurate from about 3-5 meter radius. I think this was also in the same thread talking about the government improving it though.

GPS by itself isn't accurate enough to determine what lane you're in.

Even WAAS GPS [1] (used by autopilots for airplanes) is specified to be accurate only up to 25 ft, although the measured accuracy is usually around 3 ft.

[1] https://en.wikipedia.org/wiki/Wide_Area_Augmentation_System

Autopilot has trouble with stationary objects.

https://www.wired.com/story/tesla-autopilot-why-crash-radar/

You can see the left lane line stay blue on the dash.

That's a really dangerous exit. The dark color on the end of the divider blends in with the background.

I would have thought their radar would see the lane divider. Either it didn't, or their software elected to follow the lane, or that car does not have radar hardware, or is running an older version of Autopilot (https://www.tesla.com/blog/upgrading-autopilot-seeing-world-...).

Or this is proof that their software doesn't use the output of the radar.

Without code and data, it's all just conjecture.

It was reported that the other fellow that died a few days ago also knew of this bug before he was killed. I wonder if he too died while trying to reproduce this bug, and if so, could you really hold Tesla at fault?

You know there is a deadly flaw in a product, (Imagine a car without Anti lock brakes.) But you continue to try to reproduce a deadly bug in the software. Is the maker of the car at fault?

Could absolutely be true.

The problem I see in this software issue is that the Production environment isn't fully-reproducible for testing/debugging.

We're debugging on live code on; with the ability to completely ruin lives with the wrong logic.

In comparison; Anti-lock brakes tests can be done on an abandoned airstrip.

As someone not familiar with autopilot or the programming required to "self-drive" it appears rather confusing to the car, that the left most solid-white line in the right-most non-exit lane is throwing this off. It's as if it follows it entirely.

First off, I want to say, Tesla looks bad in this case. But at what point can we start blaming the cities and states for not properly marking roads. That crash suppressor was already compressed before the accident, so did some human do the same as the Tesla?

This scares me because this looks like it would be a pretty common edge case/ something you would easily think of. How could a bug like that in such a high stakes scenario not be tested/be missed??

jeepers it seems so obvious doesn't it.

its following that white line and at a guess the profile of that stationary sudden stop at the end just seems to not fit within its target detection range.

Leave it to shitty Boston roads to nearly kill occupants and confuse Tesla's auto-pilot!

Reply on news.ycombinator.com