Hacker News Reader

Comment by alexose | Hacker News Reader

This is one reason I never understood Tesla's vision-based approach. In order to be accepted, self-driving cars don't need to be just somewhat better than humans most of the time. They need to vastly better in every situation, as you mention, to the point that they'll need every sensory advantage they can get.

I got out of a ticket once because I didn't see a "no through traffic" sign against a bright sunset. No chance that same cop gives a self-driving car a pass, nor should he.

View on HN

One of my biggest concerns about Tesla's vision based approach is that it appears to be entirely about cutting costs. Nothing about it says they actually think it is superior; the cameras on a Model 3/Y are mediocre. They were mediocre the day the Model 3 was first released. If you were going to rely on a vision system in a serious way, you'd at least invest in better camera tech. Hell, Subaru EyeSight has a significantly better camera setup, last I checked, and who looks to Subaru as a technical leader?

Someone else said it here on HN, and I think they're absolutely right -- Tesla is all about vertical integration, and this is preventing them from excelling at anything other than saving pennies. A good part of why the new EV competition is doing everything better is they didn't roll their own tech. They bought packaged solutions from companies that only do one thing, but do it well.

have you driven a tesla autopilot at night in heavy rain with no visibility of the lanes? I did, I was amazed at how good it was. I couldn't see lanes but it did and judging by the picture on the screen it saw them very well. More pixels doesn't mean better, same as more ram doesnt mean better (iOS example).

Yes, I'm on my second Tesla in fact.

I have even earned the dubious honor of having been pulled over for suspected drunk driving because I used AP on a trip home from my MIL's house after Thanksgiving dinner. I was amused by how much trouble AP was having with mild corners on I-5, the state trooper behind me was less amused. Well, at least until he established that I was sober, then we had a good laugh about autopilot and he gave my kids sticker badges. A happy ending for everyone but I didn't activate AP again that night.

Maybe the problem was that it was clear and dry? Ya think it would have been better with some road spray on the cameras?

Subsequent to that, I ended up abandoning AP altogether because it was so jarring when it would decide an overpass looked like something it should slow down for. My wife suggested that she'd be happier if I just used my foot for the accelerator. For some reason we still ended up buying another one, but after the phantom braking on day one (empty freeway, overpass, apparently this is very much now "a thing"), I guess I'll just skip AP before even starting. I wonder if I could ask Tesla to deactivate AP altogether so I could have traditional cruise control?

Sounds like Tesla should update their autopilot software to force itself to be enabled and prevent driver control at night when it’s pouring rain.

FWIW, the Subaru system is fantastic — I use it daily and it’s very predictable and literally never been dangerous. And works really well in stop-and-go traffic!

It’s not FSD, but if you use it as a driver assist, it takes a giant amount of burden off of driving in traffic. Even on a busy highway that’s going somewhat fast, it’ll just follow the car in front of you. And truly, it’s never once been dangerous.

Only catch is that it can’t “see” very long distance — if you’re going 70mph and there’s a stopped car in your lane, it probably won’t stop (at least, I wouldn’t test it!). But again, it’s driver-assist and not FSD, so you’re still expected to notice that. And within its limitations, it is really exceptional.

The Subaru EyeSight driver assist system is pretty good. But I have seen it suddenly deactivate itself several times in heavy precipitation.

On the contrary, you can hear Andrej Karpathy explain how other sensors produce noise & are too low bandwidth when compared to vision. Other full self driving systems will probably follow suit & get rid of impossible to calibrate lidar systems. How are the cameras mediocre? Do you want even higher resolution cameras? This would cause compute turmoil. https://www.youtube.com/watch?v=_W1JBAfV4Io

Karpathy claimed they worked for years and could not reap the benefits from multiple sensors however hard they tried. He seemed really convinced and does not get to me as one who tells stuff just to justify cost reductions, like Elon sometimes is carried away.

It’s also possible they weren’t able to see benefits given the processors that were available to them at the time.

It’s also possible that they were so far behind Waymo in the journey to FSD that they weren’t yet at a point where multiple sensors would make a significant difference.

This is my read on it as well. I listened to the same interview (it was definitely on the front page, if not at #1 for a while). I stepped away feeling like Karpathy had described a lot of good business reasons for not using other sensors, but not a lot of good technical reasons. Sensor fusion is hard, yes, but maybe not harder than perfectly re-projecting 2D pixel images into 3D vector space.

Just my interpretation, but it felt to me like a hail mary because they were fully committed to being the first mover. Waiting around for LiDAR prices to come down would have meant that Waymo would have beat them.

I always thought it came down to two reason, costs and looks. Telsa has to sell a car people want to drive daily. They can't have a bunch of lidar sensors on their cars no one would buy them even if they could drive themselves. Also they would most likely cost a lot more due to lidar sensors not being cheap compared to normal cameras.

Unlike Waymo who doesn't care about selling cars to people to drive daily. No one going to care that the taxi they are taking looks ugly as long as it get them to the place they are going for cheaper. The cost is also a less of a factor due to them being able to produce an income by charging people to ride in them.

> They can't have a bunch of lidar sensors on their cars

Why not? There are production cars with lidar now that isn't the big spinning thing on top of the car.

Then something else was the bottleneck at the time. It is very easy to prove that some sensors in some situations will be able to perceive things that other sensors cannot. In those situations the additional sensors are crucial first steps. I would guess the bottleneck is shitty reliance on statistical machine learning with a long tail of unhandled edge cases. Each case very uncommon, but in aggregate a very important sum.

You are misunderstanding the situation. Karpathy claimed that sensor fusion of radar, sonar and vision isn’t working well. He made no such claim about Lidar. Lidar is the sensor that is the crucial difference between Waymo and Tesla's approach to self driving.

The reason he claimed sensor fusion was not working well was due to vendor versioning. He claimed the same sensor from different manufacturing batches performed differently and thus needed to be re-characterized, which then has follow on effects in various math models. Multiply this by many sensors, and the need for replacement parts inventory for a decade or two and the problem becomes intractable was his claim on his most recent appearance on the LF podcast.

Almost sounds like a supply chain/manufacturing problem than a software problem.

Wasn't the Summon feature downgraded when they moved to a vision-only approach? One YT video I saw compared a version 1 and a more recent version and the vision-only wasn't able to do as much. Contradicts the idea additional sensors do not add value.

I think the idea is less "additional sensors don't add value in various scenarios" and more "we are 100% certain a vision-only system can perform well on existing road infrastructure; we are not sure about sensor fusion systems".

The problem is, in reality, we have Tesla vehicles that sold with other sensors that are now disabled, and those cars are now very difficult to drive with any automation enabled, even simple cruise control, due to the limitations of vision-only driving. And this is leading to them slamming on brakes at inopportune times, blinding oncoming drivers with the bright headlights, etc. - issues that did not happen when other sensors were used.

In reality, we have Tesla vehicles that sold with other sensors that are now disabled, and those cars are now better to drive with automation enabled than with the extraneous sensors, even simple cruise control is better and phantom braking is actually decreased vs spurious radar returns that existed previously, due to the limitations of low resolution radar. Auto high beams is now better at not blinding oncoming drivers with bright headlights, etc. - issues that happened all the time when other sensors were used previously.

How is your experience the direct opposite of the parent post, given the same change?

Believe it or not, if you spend enough time with a Tesla, you quickly realize that actually detecting things is a solved problem.

The thing they need to improve and are doing so rapidly is actual trajectory policy calculations.

And that’s not going to get better with more sensors seeing the same things it already sees.

This claim is not true.

As someone who drives a Tesla with the FSD beta, the vehicle has been getting progressively better since 2018.

It’s drives smoother and brakes more predictably dive they stopped using the front radar.

This claim is not true.

As someone who owned a 2019 Tesla and who owns a 2023 Tesla, the older car had better autopilot. It was starting to degrade in 2020. The new one is worse. Phantom braking was quite rare in 2019. The very first night I drove home the new car, it phantom braked on a lonely, empty stretch of I-5.

I want the radar back.

FSD stack is disabled on highways. You are using the years old code. Beta v11 when it comes out will enable OP's FSD referenced improvements for highways.

I used to mock my relatives for not wanting to put their kids in a Tesla. But yeah, I'm not subjecting my family to beta software for a critical safety feature, nor should anyone else.

I definitely have some relatives that I would trust less than Tesla to drive with my kids.

If you're referring to the Lex Friedman interview, at the start of his answer, he mentions it was a cost-based decision. And that radar/ultrasonic wasn't worth it for them, due to the additional time it took. Not that it wasn't helpful, just more effort that could be better spent elsewhere.

He clearly states that extra sensors "contribute noise and entropy into everything. And they bloat stuff." https://www.youtube.com/watch?v=_W1JBAfV4Io

Essentially that trying to utilize multiple sensors cripples any progress (given that resources will never be infinite).

It's too bad that none of their competitors, who all have better results, disagree.

While technically true that extra sensors contribute noise, surely with the proper programming they should be helpful. You just need to weight the information from you extra sensors appropriately based on your confidence the signal is accurate.

For example, if your lidar sensor is 99.999% percent sure there is an obstacle in front of you, surely it's helpful to take that information into account, even if it is a tiny bit uncertain/noisy.

Wouldn't that be more related to parallel processing power and throughout capabilities than feasibility?

More data in any situation where bandwidth is already maximized will lead to entropy and noise. However, if the capabilities were there to process all of that data in low latency scenarios with headroom to spare, surely adding additional sensors and data points would lead to a more complete model of spatial awareness for the car.

That's all hypothetical and reliant on ignoring the realities of running a business and tech development lol

A bit like our discussion here. More standpoints can ideally be merged into a coherent, more complete picture. The key is to sort out disagreements as coming from different backgrounds and biases and everybody admitting she is not 100% right. Otherwise it is a quarrel of stubborn knowitalls that can't agree.

Fusion is hard. As hard as getting humans to agree. Been there. In both situations.

And of course you can concentrate on improving one echo chamber, err single sensor. But you can never come past its fundamental limitations.

I also never understood why we had to use "vision" approaches that have the same visual spectrum as what humans see. Any sort of sensor on a device is already synthetic, why limit the spectrum that you attach it to? Should use light sensors, sound sensors, gps, everything.

So far nobody really cared how road markings, signs etc look outside of the human visual spectrum, so there's likely a lot more variation there. Both in terms of how things look like when new, and in terms of acceptable wear.

Right. But that is in designing the existing roads. I'm talking about the cars. And I'm specifically asking why not adding more options? I don't mind a camera being part of the solution at all. Gives an obvious path to human labeling of training data. But, why not have more?

We have a good idea how all this stuff looks in the infrared spectrum, and that could help a lot in the sensor fusion.

And even then, the best cameras today are still a long ways off from matching human eyesight. Tesla's cameras are not state of the art, either.

I wouldn't be shocked if the best cameras are better. :)

Point stands that the ones being put in the cars aren't.

The best cameras have much lower dynamic range than eyes (particularly in low light), and much longer times to refocus. Not to mention higher feedback latency when focusing on a particular object: eyes and brain are very tightly connected.

The eye is an incredible piece of machinery, specifically for dynamic processing like this.

I meant that mostly tongue in cheek. That the best camera ever produced may be better than our eyes. I wouldn't be shocked if such a thing exists. Probably target large and would be way more than a single sensor camera, though.

And agreed that eyes are impressive.

Right so to approximate the capability of human eyes you would need multiple coaxial cameras with varying apertures. Then stitch the images together in software.

Tesla May Be Adding A Radar

https://www.forbes.com/sites/bradtempleton/2022/12/12/tesla-...

Archived: https://archive.vn/20221212143745/https://www.forbes.com/sit...

They are standing firm on the vision-only approach because it is the correct approach. FSD cannot be perfected unless the Tesla team puts 100% exclusive focus on perfecting vision based models that don't have lidar as a fallback. Tesla can only use lidar again for redundancy only after vision is fully solved problem.

https://tvtropes.org/pmwiki/pmwiki.php/Main/BurningTheShips

Combing multiple sensory inputs is a hard problem in itself. Say Tesla uses LIDAR with vision...they can have a great vision experience, a great LIDAR experience, but fuses those two experiences together can create lots of problems that wind up being worse than either by itself.

So the beatings will continue until morale improves, eh?

This is like saying we need to get to the moon using steam power and ONLY THEN we can improve by using rockets. What if it's completely infeasible to solely use vision for FSD?

Even humans use audio processing to augment the vision.

Reply on news.ycombinator.com