Tesla FSD Beta 10 veers into pedestrians with illegal right turn
twitter.com
twitter.com
I'm starting to agree with George Hotz about making driving chill. Highway driving is chill. Stop and go traffic is chill. There's nothing chill about letting a car drive in these situations (pulling out intersections, right hand turn on red, congested streets with people everywhere) and I'm used to my car driving itself. There's nothing chill about your car having unlimited torque on the steering wheel where it can do something gnarly. Making your car do these maneuvers has the opposite effect, making you more stressed rather than taking the stress away.
I'll stick to open pilot for now. I'd rather just drive in these situations. With Tesla FSD you already have to have your hand on the wheel, might as well put your foot on the pedals.
But allowing live-testing auto-drive technology on public roads is beyound that. Police should just pull over anyone who turn this on and is on the move and revoke driver driving license. Simple as that.
"Oh, but it works for me..." is not how public security standards are created.
Before I go on, I'd like to digress and come out as a Waymo fanboi. I've been that ever since I read about a demonstration involving the software developers walking in front of the car, blindfolded. I admire that attitude, I really do. They put their feet where it matters.
But back to Tesla. What other than public roads is similar to public roads, in the senses necessary for testing driving capability? I know Waymo tests its software on recordings of public roads, but those recordings were made on public roads too. Maybe Tesla does the same, I don't know one way or the other. It's not an answer, because this approach doesn't avoid public roads, it only extracts more value from each test drive.
It would be possible to set up a test area. Maybe rent one of those "towns" the film studios have, and use stunt pedestrians, also from the film studios. I worry that those stunt pedestrians wouldn't be really like pedestrians, though, so this too doesn't replace street tests, with the real variety of real people.
These things can supplant public tests. It would be possible to regulate this and say, "software may be used for tests on public tests only when the same version of the same software has been run on one million km of recorded test drives" or such. But as far as I can see, not replace.
Recording a human driver on public roads is very different from testing a novel software product on public roads.
But suppose it was collected by a human. In that case, the first time the software is run completely (all sensors, all software, everything run together) is in the final production deployment. Is that better than testing the very same software earlier, in a beta setting which is identical to production except that sensor data is collected for the development process?
Yes, killing pedestrians is not perfect.
I mean, seriously. You are too fucking optimistic guys. Go back to the drawing board. If your program can do that you are way off from the target.
You can't write software on which innocent lives depend like you would write a startup AI showcase.
I am curious how many people died on the first airplanes and commercial airline travel ( from 1900 - 1950's )
Do you suggest that the government should have made commercial airline travel illegal until there were zero accidents?
And your analogy is imperfect. How many people died on the ground from airplane crashes from 1900-1950? Probably, not very many. (Why does the difference matter? The passengers got on the plane, knowing - at least approximately - what the risks were. The people on the ground didn't voluntarily accept any risk.)
It is just a consequence of billions of cars making trillions of trips.
But at the very least we should have a clear, reliable solution for all the easy cases. Let's summarize:
- clear, sunny day -- check, - but not blinded by the sun -- check, - slow speeds -- check, - clearly marked intersection -- check, - pedestrians not acting stupid -- check, - not trying to avoid hitting another vehicle -- check.
Just make sure under no circumstances the car is allowed to turn into a path of a pedestrians like that.
And if it can't reliably detect pedestrians or their path in perfect conditions like these I can only conclude it is far from being ready.
They posted this:
> Yes, killing pedestrians is not perfect.
In a thread about a video in which an autonomous vehicle did not kill any pedestrians.
That because it drove slowly and the supervisor took IMMEDIATE control of the car. But it looked like it had every intention to drive into these people.
> Do you suggest that the government should have made commercial airline travel illegal until there were zero accidents?
No. I suggest that government should ground commercial airplanes when they make stupid software mistakes that kill people.
Oh, actually they already do: https://en.wikipedia.org/wiki/Boeing_737_MAX_groundings
Would you have the same opinion if a self driving car killed your loved one?
You might also notice that while pedestrians and other vehicles are displayed on-screen, not a single one of the solid concrete pillars are displayed. That car is completely blind to them, and at any moment could have made a sudden LEFT turn into one of them.
Here in Washington I see people getting that wrong all the time, usually in the direction of stopping when they do not have to.
The rule is that when a school bus has its flashers on and/or its stop sign out you stop unless:
1. You are going in the opposite direction, and
2. The road is either a divided road (separated by a median or a barrier) or has at least 3 lanes.
I see people going the opposite direction often stop on three or four lane roads.
(The reason you do not have to stop on a 3+ lane road or divided road is that the law requires that if school has students from both sides of such a road the school must have separate bus stops on each side, so that no child has to cross such a road. School bus stops can only serve both sides of a road if the road is 2 or fewer lanes).
(I've not been able to find a definitive answer on whether or not a road with one lane in each direction and a center turn lane counts as a 3 lane road).
Who then believes it will even be this decade?
Although implementing might be just as hard as self driving in the first place.
At least if it would pick a damn option it'd have a 50/50 chance of picking the right one.
https://twitter.com/28delayslater/status/1437866328950517768...
https://twitter.com/BS__Exposed/status/1437367617468129280?s...
https://twitter.com/TaylorOgan/status/1437385266046586890?s=...
https://twitter.com/bagguley/status/1437328015789367297?s=20
https://twitter.com/TaylorOgan/status/1437405124004089863?s=...
https://twitter.com/BS__Exposed/status/1437043592560791552?s...
https://twitter.com/bagguley/status/1437048381042475008?s=20
https://twitter.com/BS__Exposed/status/1436806907663159299?s...
The Autosteer seems to work decently well on highway without hover hands, but it does require no sharp turns and any traffic or stop and go isn't what it was built for. I can def. see it nice when doing longer road trips.
That said, I've turned Autosteer on several times and maybe 70% comfortable with it. It does like to hug the right side of a lane, and when I'm in the very right lane on the freeway with on-ramps and exits, there have been times I didn't want to take any chances and assumed full control of steering, or else it drifts into the exit lane and gets back into the right lane, depending on the road lines and how they're drawn. This is with me having knowledge of how computer vision works - I wonder how anyone w/o tech knowledge can put their full trust into these things or to manage the situations correctly.
So even w/o the full FSD feature, I think there's still some work to do on the Autopilot autosteer (it could be a subset of FSD, maybe different ML models all together underneath) - and city driving FSD probably needs even more work.
For context, these pillars in the middle of the road are handled poorly by Tesla FSD and this guy is repeatedly stress testing the system by forcing it to drive there, for YouTube videos. The latest FSD build is the first one that actually recognizes the pillars as obstacles; earlier builds would simply plow into them. What I'm speculating happened here is the new obstacle detection failed and decided that the road ahead was blocked, so the path planner decided it had to turn right despite that not tracking the planned route.
As far as the pedestrians, it looks to me like there was plenty of space to turn into the near lane without hitting them, and the pedestrians shouldn't have been starting to cross while the red hand was flashing, but of course a law abiding driver should yield in this situation. I expect many human drivers would choose to turn anyway as long as the pedestrians were far enough away as they appeared to be. But FSD should have stopped.
> What I'm speculating happened here is the new obstacle detection failed and decided that the road ahead was blocked
Know what would be able to tell if the road ahead is blocked? Literally any kind of three dimensional sensing that isn't just single frame image object classification. It's a real shame that no one knows how to do that...
As for lidar, well lidar isn't perfect either. You might get the same behavior from a lidar car if a trash bag happened to be blown into the road ahead by the wind. All types of sensors rely on machine learning to classify their observations. Lidar isn't a shortcut that eliminates that machine learning dependency.
Finally, calling what Tesla is doing "single frame image object classification" really betrays your bias here. That hasn't been an accurate description of the system for a long time.
Better than predicting would be enough cameras to get a stereo baseline. But I'm glad at least that 6 years after launch they're finally maybe sort of thinking about it poorly.
The problem for Tesla is that they already sold FSD on cars that have the existing camera setup. I think there's a good chance that they will get to a point where they realize their existing cameras aren't good enough and will have to either come up with retrofit kits or refund owners of existing cars with FSD.
I think it's less realize, and more being forced to acknowledge
I might think differently if the moon hadn't turned into a traffic light a few months ago. But it did. That kind of failure only happens if you aren't sensing environment geometry.
Many of their error scenarios very strongly mimic what you'd get if you looked at a single frame, identified things in it, and then used the vehicle's own motion to hallucinate those things moving by zero-update dead reckoning.
The truck carrying traffic lights is another fine example:
Maybe we need a new catchphrase CAI comprehensive AI as an intermediate goal to AGI.
But then: maybe a car needs to understand the motivations of pedestrians and human drivers in order to predict their movements. And we are almost back to AGI requirements
I.e. pedestrian at corner sees car and stops, driver sees pedestrian and proceeds into turn because pedestrian is stopped at corner, pedestrian doesn’t wait and starts crossing while driver is still turning because they’re in a hurry, driver panics and slams brakes. Just seems like life at the corner of an intersection in a busy city to me?
The pedestrians had just started crossing, so it was in the wrong, but barely. It's the sort of manoeuvre that you see human drivers do all the time.
>it didn't "veer" either, it performed a normal turn.
The definition of veer is "to change direction". This did that. Stop arguing semantics.
>The pedestrians had just started crossing
I fail to see how this matters at all. Even if the pedestrians hadn't started crossing yet, and were still on the sidewalk, it would have been wrong for the Tesla to make the turn without yielding to the peds.
Contrast it to "running into the pedestrians"...
the entire point of a beta is that it's supposed to be production grade software?
"You must yield to pedestrians entering or using a crosswalk in your travel path."
At least one pedestrian was unambiguously "using a crosswalk" in the car's travel path, and the other one was entering the crosswalk as well. I wouldn't consider this a marginal case.
I think Teslas are otherwise great cars but I would never use FSD in a million years. I think it should be banned on public streets until it's gone through testing that is far more rigorous.
when it comes to hitting pedestrians, i dont think you can shade that in terms of "barely". either it is or isnt
It was on a course to illegally cut off some pedestrians.
I see the display on the left side, which shows the car's trajectory, and it shows it's going to continue going straight and then suddenly switches to a turn. I would expect it to show that the turn is going to happen sooner, not suddenly switch from straight to turning.
[edit] I linked to the law from the wrong state. Here's the right one:
https://app.leg.wa.gov/rcw/default.aspx?cite=46.61.305
RCWs > Title 46 > Chapter 46.61 > Section 46.61.305
(1) No person shall turn a vehicle or move right or left upon a roadway unless and until such movement can be made with reasonable safety nor without giving an appropriate signal in the manner hereinafter provided.
(2) A signal of intention to turn or move right or left when required shall be given continuously during not less than the last one hundred feet traveled by the vehicle before turning.
Every other state has similar laws.
The Tesla was not signaling before or during that turn. That on its own makes the turn illegal.
https://app.leg.wa.gov/rcw/default.aspx?cite=46.61.305
RCWs > Title 46 > Chapter 46.61 > Section 46.61.305
(1) No person shall turn a vehicle or move right or left upon a roadway unless and until such movement can be made with reasonable safety nor without giving an appropriate signal in the manner hereinafter provided.
(2) A signal of intention to turn or move right or left when required shall be given continuously during not less than the last one hundred feet traveled by the vehicle before turning.
Do we think that every state in the US doesn't have a similar law?
It's baffling.
You mean "making the turn illegally"? Potato potato.
This is Seattle, under the monorail (5th Ave).
> (1) The operator of an approaching vehicle shall stop and remain stopped to allow a pedestrian, bicycle, or personal delivery device to cross the roadway within an unmarked or marked crosswalk when the pedestrian, bicycle, or personal delivery device is upon or within one lane of the half of the roadway upon which the vehicle is traveling or onto which it is turning. For purposes of this section "half of the roadway" means all traffic lanes carrying traffic in one direction of travel, and includes the entire width of a one-way roadway.
The street it was turning and/or veering into is a one-way street abd so the entire width counts as "half of the roadway" and thus any pedestrian in the crosswalk counts as being "upon or within one lane of the half of the roadway upon which the vehicle is traveling or onto which it is turning".
You can only drive into the crosswalk if it is a two-way road, no pedestrians are in the part of the crosswalk on the side that you are driving, there are at least two lanes going the other way, and none of the pedestrians on the other side are in the lane on that side that is adjacent to your side.
But there have been plenty of failures, and remember that the only people who have the beta are employees who are presumably under an NDA, and devoted fans who have publicly said they don’t want to post the disengagements because critics “take them out of context”
Other failures:
https://twitter.com/28delayslater/status/1437866328950517768...
https://twitter.com/BS__Exposed/status/1437367617468129280?s...
https://twitter.com/TaylorOgan/status/1437385266046586890?s=...
https://twitter.com/bagguley/status/1437328015789367297?s=20
https://twitter.com/TaylorOgan/status/1437405124004089863?s=...
https://twitter.com/BS__Exposed/status/1437043592560791552?s...
https://twitter.com/bagguley/status/1437048381042475008?s=20
https://twitter.com/BS__Exposed/status/1436806907663159299?s...
The Tesla FSD would've caused a t-bone in the first two clips, and a head-on collision at decent speed in the third clip. Yikes.
Probably regional though.
It depends on where you live. In downtown Vancouver there were arrows at each intersection telling you what directions of travel were legal.
Moral of the story, do you think most humans would do a better job at this intersection than the Tesla did?
Or have we truly reached the point now where we expect "stable" software to be of the same calibre as beta software so nobody makes a distinction between the two?
The car should be much safer before it goes on the road, regardless of test status. This was a near miss and should be taken extremely seriously by highway safety regulators.
>Or have we truly reached the point now where we expect "stable" software to be of the same calibre as beta software so nobody makes a distinction between the two?
I expect every vehicle on the road to meet a safety standard. I do not accept "beta software" tests on public roads if this is the kind of mistake they can make.