Tesla FSD Beta almost causes a head-on collision [video]

The trick is its "Fully Self-Driving" TM. Which doesn't mean what the rest of the industry means by that.

This was this 10 of May 2022

https://www.ft.com/content/50d2861e-fdeb-487c-b052-f2021558e...

That’s not what ditching means.

> even if said car on average kills fewer people

That is only based on butchered up statistics. You would get very good statistics with a goddamn robot vacuum on a motorway, because it is a relatively uneventful driving environment. Guess where people will take over the wheel? In dangerous situations, basically self-filtering any negatives from the statistics. Very convenient.

>but doing that in the dark is really only possible with LIDAR at the moment.

Which seems a good enough reason that we need to add regulation that these vehicles must have multiple redundant sensors that can operate at any permutation of sunlight/night/rain/snow/fog.

Radar can also see at night, but has poor resolution. Sonar is good at looking through fog but may be unreliable at highway speeds. Lidar really seems like the only viable option here. Maybe even lidar + radar.

I think it's interesting that Elon Musk always says we don't need lidar because it's a visual world. Lidar is a visual sensor. Just an active one.

A human in my country is supposed to drive in such a way as to be able to stop in the distance that they can see road is clear. So it's not that crazy to ask from Tesla that the car slows down if it can't judge to road to be free, that's exactly what we ask from human drivers.

Not all humans manage to do that, and they're liable if they crash into some other car. Loose their insurance and cannot drive anymore if they do it too often.

> You can't blindly assume your path forward is default-safe. Or you can yolo and risk being part of the 40k dead and countless maimed every year.

Or worse, be the one responsible for killing or maiming others.

Seems like stereo from cameras in the upper corners of the windshield would provide valuable depth perception. But what do I know...

It should be made illegal ASAP and those who bought it be refunded

Either way, you don't own it.

To a private entity owned by a scammer.

But don’t get me wrong, I can emphasize, I had plenty of bad purchases as well..

You might be thinking of unlimited supercharging - that was initially tied to the car, then they started removing it if the car was moved to another account.

*only I the US.

I'm surprised the FSD is even legal in the EU. Maybe not for long...

FSD gets unenrolled. But the new car owner can re-enroll for no cost, just clicking through some warning screens telling you to pay attention.

You won't have to pay for FSD again. Stop talking nonsense. And what do you mean, 'upgrade your car'? Specifically? I am not aware of any actions that remove a previously paid FSD entitlement from a Tesla.

Yeah, I might not be remembering the story correctly. It might have been collision avoidance rather than the FSD portion of their firmware.

I still don't think that FSD is really for prime-time in any capacity though. Seems to make far _far_ too many mistakes that users need to correct. Does that sound like Full Self Driving to anybody? :P

Maybe Tesla radar can't see any stationary objects, but others use radar for collision avoidance / auto braking and seem to do so also for stationary objects. It's harder to do from a signal processing view, but definitely not impossible.

I don't think that was the point they were making.

If you watch videos of drunk non-Tesla drivers you get the impression manual driving is unsafe.

But on a side node the long term aim is hopefully that even drunk / sleeping / 10-year old people can use self driving cars.

There are 30,000+ car accident related deaths in the US alone every year. There are 365 days in a year. That works out to an opportunity for roughly ninety videos of fatal car accidents every day. Watch the ninety videos of people dying before watching the one video in which no one dies and maybe you will be able to understand the argument better. The actual raw data - the safety statistics - are very clear. Human + AI is safer than human alone and Tesla's cars have the lowest probability of injury given an accident of any of the cars on our roads. The statistics are so opposite your thesis. The reason you have your thesis is because you have been misinformed by the selection bias.

Hmm, the view of the screen isn't clear enough to tell when exactly the car appears. An eerily familiar problem, lol. But that's kind of beside the point. Here's a screengrab of slightly sooner, at 8:05.

https://i.imgur.com/Hj6glgu.png

The car made a plan to cut across the yellows and then continue straight down the opposing lane. It shouldn't do that whether or not it sees traffic up ahead.

It's been a common issue in these systems that they seem to lack object permanence

I have personally witnessed low angle red sun give artifacts on a Velodyne Lidar. It was not repeatable another sunset, to make it worse.

Well, public transit solves this by having the vast majority of riders stand and pack very tightly during these times. No reason you can't be cramped into autonomous EVs, other than people don't want to do that.

This is the issue with comparing transit to cars - the amazing capacity numbers touted by eg subways are based on great rider discomfort.

The total number of cars is still a problem - they take up street parking, they require large parking structures to store during the day, etc, etc. You should think of autonomous EVs as red blood cells - constantly circulating and carrying useful loads everywhere, not sitting idle 90 percent of the time.

I live in a city that is awful for cars (Buenos Aires) and has kind of serviceable public transport (it's the method of transport for most people). Yet people that can afford to own a car will do so. And they'll use it whenever the situation allows it (mostly pparking related).

Remember that owning a car doesn't ban you from using the subway or whatever if it's a better choice for that trip.

And public transport will never have the privacy that a car has. Comfortable heated seats. Setting the AC to your liking. Blasting your music. Having a quiet conversation with your partner. Not worrying about others people COVID or whatever. People selling you stuff or crazy in general. Leaving from your driveway and arriving at the destination's doorway. Which is a big deal if it's raining, if you're with your elderly parent, if your carrying something heavy, etc. And if you're a nocturnal beast like me, cars are so much better in general.

BTW I much preferred living in LA when I lived there, to the few stays I had in NYC Manhattan, and to Buenos Aires.

> But the reason we find them convenient is that, at least in most of the US, cities are built for cars.

A subtly better way to frame this is to say that American cities were destroyed for cars. Streetcar tracks were ripped up. Roads were widened at the expense of sidewalk. Neighborhoods were demolished and replaced with parking lots, freeways, and car dealerships. Many municipalities stopped maintaining their core inner cities and bet the house on big-box stores on the outskirts. It cost hundreds of billions of dollars collectively to subsidize the car in the U.S., which had the side effect of cannibalizing passenger rail and metropolitan transit, as well as building places that are prohibitively dangerous to cycle or walk in.

One illuminating exercise is to find photos of Houston from the 1920s and compare them to the 1970s. You would think that someone had carpet-bombed the city.

False impression and common. We don't need to guess here. There was an engineer talk at the Computer Vision and Pattern Matching conference in 2021 in the Workshop on Autonomous Driving by Andrej Karpathy senior director of AI at Tesla. In the latter portions of the talk he dives deep into why they choose to go all in on video with compelling empirical results showing it to be far more accurate than vision + radar.

https://news.ycombinator.com/item?id=31414374

They do end up doing sensor fusion, but not when you would think. At training time they use radar still, because at training time you can do things to correct for radars errors, for example, you can use hindsight to reason about what happened. At inference time, the uncertainty of hallucinating radars is a real risk to safety. It says things aren't there when they are. It says things are there when they aren't. That makes it a lot harder for the car to plan out the next actions and promotes very dangerous action - like slamming on the brakes - which can surprise other drivers and cause accidents.

If they were Tesla only problems, why did other companies feel the need to advance the state of the art in their radar hardware? They did so because poor radar equipment wasn't a Tesla only problem, but was a problem that many identified. Poor radar was not a Tesla only problem, but a common problem.

Now the world state advanced to a world in which WayMo had more advanced radars and Tesla did not have more advanced radars, but instead stopped using bad radars. Both companies however made the decision to abandon the radars that they previously had. Both identified getting rid of their former radars as the correct solution to the problem of radars which did not meet their needs. So it isn't the case that it was a Tesla only problem and it isn't the case that the two companies had terribly different ideas about how to solve the problem. Both companies improved their car by relying on technology to make up for shortcoming in radar systems. The difference was in how they did it. Tesla did it by devoting more effort to the vision stack whereas WayMo did it by devoting more effort to hardware improvements.

In the video you dismiss there are segments that show Tesla's vision-only system successfully identifying distances despite the presence of fog. They also identify things that are stationary. They also identify things that are moving. So the things you cite for supporting your thesis that Tesla failed aren't examples of things that Tesla is doing that WayMo isn't. They are examples of things that both cars are doing. That means they are worthless for making a claim of superiority.

Having failed to establish superiority you move on to claiming that Tesla wanted to cut costs and that this was their motivation for doing this. This is cast in a bad light as if cutting cost harms safety. In reality training data is precious and collecting it improves performance. Cutting cost allows greater volume of production by lowering the capital cost of production. This results in more sales. This results in more data. The net result is a better training set, improving performance. You aren't even correct that cutting costs actually results in less spending on solving the critical problems: cutting costs leads to increased sales which leads to increased revenue and in turn increased profits which leads to more capital for development, not less capital for development, which means more capital gets allocated to solving self-driving problems. As such cutting costs has the impact of increasing the amount that is spent on researching the problem as time goes on.

The big difference in their choices isn't that Tesla is approaching it in a bad way and WayMo in a good way. It is that WayMo is getting funding from projects that aren't itself. It can afford to make bad decisions in terms of cost-effectiveness when it thinks that it improves the probability of eventual success. This sort of spending model has historically been simultaneously underwhelming and overwhelming in what it produced. Overwhelming because it does things like get us to the moon, but underwhelming in that it does so in such a cost ineffective way such that we don't return.

This is shaping up to be the difference between WayMo and Tesla in practice. WayMo handles driving in fenced off locations well! This is the overwhelming awesome thing it does. However, at the same time, it has extremely low production volume and doesn't support all cities. This is like how we got to the moon, but then didn't get to keep going there. Meanwhile, Tesla has literally millions upon millions more self-driving miles.

However, this actually works out in practice to WayMo harming public health in comparison to Tesla? Why? Well, self-driving plus humans is already better than humans driving alone. WayMo has technology that could save lives to the tune of some fraction of the ninety people in the US who die every day in accidents not dying. WayMo isn't letting that technology save those lives. Instead, Tesla is letting a similar technology save those lives. As Tesla scales it captures more of the market, it stops more of those deaths. Thus in terms of their net benefit in terms of death reduction, WayMo actually loses by far. Every moment they don't scale while Tesla does scale, they lose some more. The only hope for them catching up is to achieve greater scale than Tesla which is going to give them a cost effectiveness problem or alternatively they need Tesla to mess up in a truly massive way that forces regulators hands - something that regulators and Tesla don't want at present, because the current trend is lives being saved from human errors.

Yeah, back then I also thought that a leader-follower convoying system would be implemented first (with local drivers for the last mile problem), because that system made sense and was straightforward- with good logistics software (to make sure that the leader is driving the farthest distance or connecting multiple leaders together) you could easily save 2/3 or more of your driver costs.

Then a few years ago I talked to an actual software consultant to shipping companies, and asked him why none of that had happened in over a decade. He felt was that the companies were too fragmented for that. Most shipping companies are small companies, where basically the owners wife keeps a list of all of their shipping loads and destinations. Maybe Walmart has the access to capital to acquire that technology and the scale to profit from it, but he felt that no one else could. Which was why he left that field and went into industrial robotics, where the business case can actually close.

Worse- it was a 15 ton Stryker ICV versus a mere 2.5 ton Humvee.

I meant it as a joke (hopefully), but a technical reply to your question would be "violating the non-disclosure agreement": https://www.theverge.com/2021/9/28/22696463/tesla-fsd-beta-n...

Humans crash cars pretty regularly, but I bet if we had lidar/radar vision we could do a bit better too.

Radar gives false positives. Pretend you are a computer for a second. You get told by radar that right in front of you there is an obstacle. You are five feet from crashing. The only option to avoid it would be to slam on your breaks. Meanwhile the vision system tells you that you are on a freeway, there is a car directly behind you, and there is a bridge ahead of you.

Do you a) slam on the breaks or b) drive under the underpass that was producing the false senor reading?

You've already declared your answer. You choose option a, because you think radar is amazing and you think people who think otherwise are naive morons. What happens next is that you slam on the brakes, surprising the person behind you. They slam into your vehicle. Their child wasn't wearing his seat belt. He flies forward, slamming through the window. His brains splatter the pavement. His body rolls without his brains into another lane. A horrified person to your left swerves to avoid hitting the kids body as they drive by, plowing into another car.

Congratulations. You are a genius. Everyone else is naive. Thank you for playing murder innocent people.

Of course, this isn't what actually happens. What actually happens is that your decision results in a sudden break, but the person at the wheel recognizes this is a mistake and presses down on the pedal. Your decision making ability is taken away, because you are a moron. They do so, because they trusted their vision system more than they trusted your stupid radar based decision. So the error correction mechanism that stopped your murder attempt? Vision.

Thank God for that, but the person in the vehicle is annoyed. They report the issue, not liking that bridges consistently produce that behavior. Tesla investigates. They realize that the radar sensor is producing false positives. After empirically validating that removing radar is better at driving in this edge case they roll out the improvement. Tesla removes your ability to decide to kill people because you are too obsessed with radar.

Later, even as the empirical results show that self driving is now better than human driving in terms of safety, partly of course due to it being human+computer driving now computer alone, a person names knodi123 goes online and calls people naive for thinking that vision is preferable to radar.

He gets asked this question. If you were the driver of that car, which would you rather trust? The car's decision made on the basis of radar? Or your own decision made on the basis of vision?

In my other reply I tried to say you were wrong in a way that was rude, which I regret, so I want to try again in a more respectful way.

Initially, it was the thought of most people who thought about the problem that more sensors would be best. This was what Tesla thought, not just what you thought. It was what I thought. I've read Artificial Intelligence: A Modern Approach. I've read about Kalman filters and sensor fusion. I've implemented them. So intuitively, I think it makes a lot of sense that more sensors should be more effective. So it was surprising to me when Tesla decided to drop radar and it was even more surprising to me when they shared that the empirical results of dropping radar led to measurable improvements in vehicle safety and improved their accuracy in determining their position relative to other objects.

In the past I've made that claim and people have been surprised by it. It doesn't seem to be common knowledge. If you didn't know it or you doubt it, then I'd recommend you check out an engineering talk by Andrej Karpathy. This isn't a Tesla marketing piece. It was a workshop talk at the Computer Vision and Pattern Matching conference in 2021. Andrej Karpathy is the senior director of AI at Tesla. He was someone involved in this decision to switch from radar to vision only and in the talk he outlines the engineering reasons which motivate the switch.

[1]: https://www.youtube.com/watch?t=28257&v=eOL_rCK59ZI&feature=...

If you still doubt that it is a good idea to switch, I can contribute my own anecdotal experience. I have a Tesla. I drive in it. When I do, I use autopilot very frequently. The majority of my driving is done by autopilot. As such I've gotten experience which has informed me a bit about how autopilot tended to fail. One of the ways it could fail was by making me take over when going under an underpass or by breaking suddenly when doing so made little sense. This is a category of error that I have stopped experiencing since the switch away from radar.

So now we're in an interesting position, because if you recall you claim that people who believe vision only are naive. Naivety is usually defined as meaning to be wrong because of idyllic assumptions that are wrong, but which you don't realize are wrong because of ignorance. Yet the way things played out historically is that people assumed that sensor fusion was the best approach, empirical results suggested it wasn't the best approach, and consequent to that people changed their minds. In terms of the progression pattern, this is exactly opposite of what it means to be naive, because the beliefs are contingent on experience rather than a consequence of its lack.

Sometimes we have simple models of reality and they suggest one thing. Then we get experience in the real world, which is much more complex than our simple model, and that experience tells us something else. Telsa, like you, thought radars plus vision was better. They tried a model that dropped radars and the result was empirically measured as being safer.

Should we go back to the thing that we know to be less safe through empiricism? Well, if we do, but we do so without solving the reasons it was bad, then we are going to have the car making bad decisions. Those bad decisions put lives at risk. So I don't think we should go back. If we can address the root causes of why the sensor fusion approach introduces dangerous error, then we can do that, but just adding back radar? That would kill people. Probably the people directly behind a Tesla that breaks because of a hallucinated object detected by radar.

I wish I could delete my other reply, but I think the example in it is extremely important, because when people try to use the toy problems that fit in their head? They are choosing to exclude that very very real non-theoretical situation. That is exactly what the world with radar was generating as an inevitability and we need to be crystal clear on that when we reason out exactly how to avoid that type of error. Blindly saying that sensor fusion is better is really dangerous, because people can really die if we make the wrong decisions -- and because we can measure the results of different models empirically? It is blindness to say it, because it contradicts the evidence.

If you rely on radar when you shouldn't rely on radar, people can die. So the answer to half your question is that even a minuscule probability of misidentification has intolerable consequences. If you rely on radar, because you can't rely on vision, people can also die. You are driving while blind to things like stop lights. Unless vision conditions improve it isn't safe to drive. Also there are going to be the same edge cases where radar failed, but now vision won't be able to compensate for them. Since radar can also fail by failing to detect something there, not just detecting something that isn't there, this is a fatality waiting to happen. In every other case, since the error bars are lower on vision than they are on radar, you ought to be relying on vision. Therefore, since in every conditional case in which driving is reasonable, vision is demanded, engineering effort is better directed toward vision than radar.

Read my post again. I clearly articulated that there is a timestamp where it plans a bad route briefly. That is something to investigate and fix even if it didn't ultimately keep that route. It is a very serious issue and no one should diminish it.

I'm sorry that you only get your information from second hand sources and so can't understand how I can both know what you said, but also know that you are under-informed. I realize that a headline and a single screenshot may seem extremely informative to you, especially if you skim comments rather than reading them in their entirety. However, I didn't look at a screenshot. I watched the video multiple times, including in slow motion, with special emphasis on the time where the incident occurred.

I strongly feel that if you are going to try to correct someone, you should really finish reading their post before you try to correct them.

I share your concern for that timestamp and think this error needs attention from Tesla.

Which of my claims do you disagree with?

- That the vehicle planned a route that it shouldn't, but corrected that route plan?

- That the vehicle entered an area where it shouldn't enter?

- That Tesla doesn't use radar?

- That Tesla doesn't use lidar?

- That the calculations were done on GPU rather than CPU and so it doesn't make sense to think that the reason this happened was an overloaded CPU?

- That the Hacker News title is editorialized, in contradiction to Hacker News guidelines?

I think the last one is probably most contentious and you are most likely to consider it sarcasm. So I'll point this out and hopefully you'll see where I'm coming from. If you renamed the video title to, "Tesla FSB Beta does not cause a head-on collision" it would be just as accurate for the situation in question. However, it would also be more accurate for the majority of the video. As such, it is a more accurate headline. Meanwhile, the name of the video is changed from the name on YouTube. Clearly an editorial choice was made.

Was it a good one?

To really get to the heart of the matter: no one died in this video, but if we had a similar bias against traditional vehicles we would see 90 videos of deaths every single day. A similar level of antagonism would demand that more then the entirety of the entire front page of Hacker News was completely taken up with actual deaths. Every day. The actual statistics at the heart of driving safety are very clear - human + AI is currently much much safer than human alone. That is the actual reality. This headline contradicts the underlying reality and it does so because the author knew the headline would attract more attention. It was optimizing for engagement, not for accuracy. I feel this sort of approach to headlines strongly contradicts the guidelines of Hacker News.

You can use if without having purchased it - it's available on a monthly subscription basis.

How often did horses breakdown? Do you see why someone from 1890 may think it is crazy to drive a car that frequently has a breakdown vs a horse?

Similarly, 130 years from now people will ask "Why would you want to drive your self when there were self driving cars available? And someone else will have to explain that FSD required close supervision and needed interventions between 1 or 100 miles, depending on the driving situations."

Not advocating for anyone to beta test. Just pointing out the draw.

Excuse me but Tesla makes drivers pinky swear that they will pay attention 100% of the time and be ready to take over driving at the drop of a hat. Obviously you're just a luddite.

/s

It may be ethical if it really does lead to saving many lives by advancing the state of the art more quickly than it otherwise would have.

And given how many people are drunk while intoxicated or using smartphones, it seems like a relatively negligible increase in danger.

The math might be something like: police departments could increase enforcement of DUI and reckless driving laws by 1% and compensate for the total increase of danger caused by Tesla's FSD testing.