Tesla Autopilot Almost Steers Model 3 into Oncoming Train

- my city town inexplicably decided to make dashed white lines on the sides of bicycle lane sharing markers. The Tesla would see dashed white lines, which mean separate lanes going the same way and dance around trying to get into a lane but they were all too narrow. The traffic lane was double wide to account for church street parking, but not marked as such. So it the markings made a tiny center lane and two side lanes that were just a bit too small.

- on winding tree lined roads, when an on coming car first appeared around a bend it would assume the car might be going straight and would hit us. There would be a quick deceleration and dodge toward the shoulder until it realize the oncoming car was actually curving and would probably stay in its lane.

- when the pavement expands to include an unoccupied, unmarked parking lane for a short stretch the car would assume that the road suddenly got very wide lanes and it was too far to the center and jerk over.

At this point FSD does a passable job on my regular daily drives, except one intersection where something in its map is wrong, it wants to exit the right turn lane just before turning right. I don’t let it make turns in contested intersections yet, because it is a little timid and might confuse drivers. But that is improving. Just not there yet.

For exceptions, it does well finding and dealing with construction areas, garbage trucks, bicycles, and pedestrians.

I’d appreciate if they’d add “slightly dodge roadkill”. It will merrily run over the same dead squirrel four times a day with precision.

First there was zero monitoring of attention. No steering wheel torque, nothing.

Then when pushed and threatened they added it, and set the "hands on steering wheel" interval to "ONCE every fifteen MINUTES".

Then they had to be threatened to reduce that to a sensible value.

All the while, Tesla: "the driver is only in the seat for legal purpose - the car is driving itself", or "use Summon to bring your car to you while you deal with a fussy child".

So you can call them lazy edge-case idiots all you like.

But you can't pretend that Tesla hasn't absolutely fostered and nurtured this culture, because they have.

People who understand the system don't buy it because they realize it's a scam, everyone who bought it was duped. The most valuable thing at Tesla is the list of rubes they have.

Let's define autonomous as self driving. According to their marketing page[1]:

> Tesla cars come standard with advanced hardware capable of providing Autopilot features, and full self-driving capabilities—through software updates designed to improve functionality over time.

So it has capabilities of autonomous driving, which means a Tesla can be considered an autonomous car. Unless I'm an idiot for trusting what their marketing page tells me, of course.

1: https://www.tesla.com/autopilot

Imagine believing this.

edit: whoa somebody is flagging all of critical comments of tesla!

Fending off lawyers is based on the fact it it was only Musk who said it, courts recognizing that only an idiot would believe him.

A human does not need to be taught how to handle every edge case because we have the ability and intuition to think on the spot and create solutions on the fly. It's incredible how fast our brain can take in information and formulate a plan.

Meanwhile, if an "AI" has not specifically been trained against a certain variable or situation, it will have no idea how to handle it. It can't actually create a solution like a real human brain can.

Or has the tech industry beaten any semblance of self-respect out of everyone to the point it is just expected the technocrats get to use everyone as guinea pigs?

Beyond that, the way it is calculated makes it less of a safety score and more of a risk score; it can't distinguish between a person driving effectively in a risky environment (safe) and a person creating a risky environment (unsafe). Some portion of the testers are then people who actively avoid safe defensive driving techniques in risky environments to maintain their safety score. That type of person is the absolute last person you want testing an autonomous system.

This video is a great overview of FSD beta safety, I've linked to the section specifically about the safety score: https://youtu.be/sHyOL_vDQMQ?t=801

Procrustean bed, hooooo!

I do have a photo of that somewhere.

https://twitter.com/elonmusk/status/1508110810614181890

Granting you the pedantry point here, I hereby happily amend my comment to refer to "data [evaluators] and annotators", with zero impact on my point.

> So, who is being dishonest?

Not sure if you're actually familiar with the definition of the word? Being potentially technically wrong on a pt that doesn't change the overall claim is enormously different from the core of a claim being wrong (eg the implication that "supposedly important people haven't been laid off" when they demonstrably focused the layoffs on the most commoditizable and replaceable labor in the industry).

[1] https://archive.ph/i6BCD

"A Cruise-on-Cruise Crash Reveals the Hardest Thing About Self-Driving Tech": https://www.wired.com/story/cruise-on-cruise-crash-hardest-t...

In this case a Cruise car was rear ended by a Cruise car driven by a human, because their unsophisticated software makes for the car to break hard. As explained in the article:

"...That said, the rear-endings demonstrate that the technology is far from perfect. Cruise cars follow road laws to a T, coming to full stops at stop signs and braking for yellow lights. But human drivers don’t—and Cruise cars will be self-driving among humans for decades to come..."

"...The fact that a driver Cruise trained to work with these vehicles still managed to rear-end one emphasizes exactly how flawed they are..."

Congress sets executive agency scope. Not executive branch employees or judges out of force of habit.

If they want increased scope, best ask the People if that's alright, and get the People leaning on their legislators.

Bottom up. Not top down.

How many times to people have to be “almost” shot, but not actually shot, before we concede that maybe the gun has no bullets?

But this is how all aircraft autopilot systems work - they expect a pilot to be able to step in. The pilots don't leave the controls.

Isn't the safety of running the beta (the issue being discussed) a function of the safety of the driver?

Not saying these incidents aren't real, but I'd really really want confirm these from a reliable source.

Should be FTC, but yeah. This seems like a clear case of an unfair and deceptive trade practice that (literally, physically) injures consumers.

Their claim isn’t about their cars; it’s about the fact that some software can be created that gives their cars “a self-driving capability at a safety level substantially greater than that of a human driver”. Even if they never manage to deliver such software, that’s no proof such software doesn’t exist.

To be fair, failing to yield to oncoming traffic when turning left is an extremely common mistake for human drivers to make as well.

In fact, earlier in the video, the car correctly yielded to an oncoming car, the human driver overrode it, then complained that the oncoming car "cut him off"! https://youtu.be/yxX4tDkSc_g?t=494

I'm a private pilot though and what you're describing sounds a lot like the autopilots on small planes. It handles keeping course and altitude for you, so you can focus on the rest of the flight and it is indeed way less taxing than hand flying a longer distance. And like Tesla, a plane autopilot will occasionally do something weird so you can't entirely ignore it. I would think the difference though is that in a plane, nothing the autopilot does will be fatal or dangerous very quickly, so you have plenty of time to notice and correct it as long as you aren't sleeping or something. When you're driving a car though, a second of steering in the wrong direction can quickly turn into an accident, so you as the driver have to be constantly ready to immediately take over. Even in the chance is low, as long as it's often enough to be a factor, you still need that instant readiness.

I wonder if the value is more on longer drives, as you said, vs city driving. The fatigue factor driving between cities is a bigger issue, and the driving tasks are simpler with less opportunity for the car to do something crazy than in busy city driving.

Assuming you know its strengths and weaknesses. See next post, below.

I'm wondering if "reducing major components of normal driving behavior" is such a good idea?

What I'm getting at-- the delusion/illusion that the brain/mind is somehow separate from the body-- that learning can all be in the abstract without any connection to tangible reality.

How can you make throttle decisions if you don't have throttle control? Wouldn't having throttle control at all times mean you have more awareness of the machine you are controlling and can thus decide faster and act better? Because this control is already part of your thoughts?

And before i hear a "That's exclusionary", yes, excluding people unable to be safe while driving 2 tons of steel launched at high speeds is a good thing

As others have said, I cannot fathom how having to babysit an occasionally suicidal driving system is better than having no system at all. It seems perfectly balanced at the absolutely worst point of both being reliable enough that you stop paying attention, but unreliable enough that it catches you not paying attention.

If an FSD system really let me sit and watch a movie or read or book or something -- if it was that good -- then I 100% get it and would be signed up. But one that you have to babysit seems worse than pointless.

Look, there are some things I really like about your reply, and I applaud your approach to this.

My question is this: If you are the pilot, how do you communicate with the FSD to have it execute the maneuvers? What is the command interface? What is the level of "manouver" ("drive to the grocery store" vs "angle the front wheels 7 degrees to the left to initiate a left turn")

> "there were a couple of times I expected [the loaner] to stop and it didn't."

So these times, in your car, are circumstances where you expect the car to stop. Did you (as the pilot) tell the car to execute the stopping manouver?

> Your #1 job, and the thing that can kill you, is thinking that autopilot is engaged when it isn't and running a red light "

What if it is engaged and still decides to run a red light? Or to suddenly brake to a full stop on a highway so you get rear-ended?

You need to meditate on this. Hard. You're more right than even you realize. Your cars behavior will change on you overnight. Your access to charging infra, or even the capacity to access the full capabilities of your vehicle are locked away, and prone to change from day to day. You don't get to opt out. You don't get to say "No, this is fine exactly as is."

What you believe is your car, is by no means your car.

It seems dangerous to drive without doing that.

That's what I don't get. You say it's useful. That implies you're doing something you weren't able to before, or getting some extra utility out of it, yet, from my understanding, you legally speaking should be no less attentive in the operation of this vehicle than any other.

In fact, you let your family operate it in FSD? Are you really doing them any favors? Driving is hard enough without having to consciously override an additional source of errors. You may be unintentionally training them to operate Autopilot/FSD instead of driving safely in general.

There's a reason these problems stick around.

[1] https://en.m.wikipedia.org/wiki/Moravec%27s_paradox

To Musk and his goblins, that's basically just cameras. But it's okay, people keep assuring me he's a great engineer.

Or a complete fucking sociopath that's lied his way to success but hey

Because they'll need to set up an entire pipeline to get those images into the system. Meanwhile pictures from streetview is plentiful and free (both in terms of licensing costs and development costs)

>Besides if robots can do it then someone would be using them to solve CAPTCHAs.

Maybe such models are in the hands of the top AI companies (ie. companies making autopilot systems), and they're not really interested in starting a side business of captcha bypass?

The other possibility is that they're already in the hands of bad guys, they're just putting it under wraps so they can monetize it and not get it detected. For instance there's this: https://addons.mozilla.org/en-US/firefox/addon/buster-captch..., so the idea isn't too far fetched.

Neither of those conditions are the case. I'd appreciate it if people would stop trying to use statistics to gove an unproven system being tested with a fundamentally unsafe methodology with minimal experimental controls a free pass.

I do however think that it's wrong, why? Because human error is enough to deal with, couple this with now AI error. It makes society, crash investigation, designing roads way way more complicated.

Maybe if we ban human drivers 100% and optimize for cars then it would be a good idea. I just can't see cars not needing supervision for a very long time.

I doubt people are going to buy cars that might kill them by making "mistakes". We will see but I think it's going to be a hard sell. People can easily accept their own mistakes. A computer though?

Unfortunately it sucks for all the innocent people on the train.