Hacker News Reader

Comment by gonehome | Hacker News Reader

The argument was more that to do it generally you need vision to work, lidar will always be a narrow solution and redundant when you have vision.

It might end up that solving the narrow city problem with high res maps is valuable enough to tackle anyway, but it could still be true that will never solve the general case.

Of course, it could be true that Tesla was be unable to solve it also. Though at some point attaching some general intelligence to vision can probably do it. Likely will be bigger issues then though.

View on HN

Just because humans can generally do it without lidar and other sensors, does not mean that lidar and other sensors, would not improve human driving, if we could somehow integrate them seamlessly (and well reduce their price).

A car equipped with more sensors, will always be better at handling more situations, than the one without them, no matter how "smart" their drivers become.

I think the overlooked part is "seamlessly integrate". Signal to noise ratio is important, and it is not trivial to "just add sensors". I think the analogy of "too many cooks in the kitchen" applies here.

I think the keyword you’re looking for is ‘sensor fusion’, which apparently is only a problem for Tesla because the others have figured it out years ago. Perhaps there is a talent gap at Tesla or an unwillingness to invest in it. They generally avoid having to do anything that’s hard and try to look for shortcuts.

Case in point: the whole radar removal and reintroduction flip flop. It seems like they don’t believe themselves that cameras are sufficient.

I don't think anyone has "figured it out" if they still don't have unconstrained driverless cars.

That’s tangential to your original point of discussion which was sensor integration. Sensor fusion is absolutely a solved problem. Autonomous driving requires more than just sensor fusion though.

It's not tangential at all. If the purpose of these sensors is for autonomous driving, and "solving" sensor fusion hasn't gotten you there, how can you possibly say it is a solved problem? Sensor fusion (in this context) is not solved until you are using fused sensors for autonomous driving.

Why are you equating sensor fusion with solving the entirety of autonomous driving? That’s disingenuous. You understand sub problems can be solved, right? Sensor fusion is just combining different sensor inputs so you have high confidence in your object detection and you’re not “confused with disagreements” as Tesla likes to say. Self driving involves solving prediction and planning problems too, not just sensor fusion.

I'm not equating it with solving the entire problem, I am pushing back against the idea that autonomous driving can be sub-divided into completely discrete sub-problems. So I guess you could say no, I do not understand that sub-problems can be solved.

Object detection is inextricably linked with prediction which are both inextricably linked with planning.

Of course, they are linked. It doesn't mean you need to operate worldwide to prove sensor fusion is working. There is no scaling issue with it. Sensor fusion is already proven effective as companies using it are showing excellent safety records where they are operate. The Tesla complaints about "what do you do in case of disagreements?" don't hold good anymore. They just don't want to invest in get it working (but are now forced to as they re-introduce radar in their vehicles).

What evidence do you have that there is no scaling problem with sensor fusion?

As you point out, the companies that have "solved" it don't have scale.

Sensor fusion has nothing to do with scale. They are merely algorithms that combine multiple real time inputs into a single low level representation. This process remains the same whether you operate in San Francisco or worldwide. How they are interpreted is what differs from a Waymo to a Cruise to a Tesla. Waymo/Cruise get richer input they can feed into their downstream perception algorithms. Look at Waymo’s object detection benchmarks against Argoverse, KITTI and Waymo Open Dataset for their performance. They top most of them.

They don’t have scale yet for different reasons (adverse weather driving, long testing cycles, costly operations setup). “If they don’t have scale, that must mean sensor fusion doesn’t work” is quite a ridiculous argument.

If you think sensor fusion is not possible, then you must also believe Tesla’s attempts at re-integrating radar will not work and that they won’t achieve their stated goal of FSD.

I never said I don't think sensor fusion is possible. My only argument has always been "adding more sensors does not mean you will have a better self-driving car". Richer data does not mean better data. It does not mean more actionable data either. All it means is that you have more data points. I absolutely agree that this can be a good thing, and probably is in the case of self-driving.

But maybe it's not. Maybe the additional flops required for sensor fusion would be more useful if given over to the planning part of the problem. After all, you don't have infinite cycles. This is what I mean when I say it has not been proven at scale. The edge cases matter (arguably they are all that really matter), and "solutions with more sensors work better at object detection in artificial benchmarks" is not nearly as convincing to me as it seems to be to you. Again, object detection does not exist in a vacuum, and a solution that generally behaves better in a very constrained environment might actually behave worse when you throw all the edge cases at it and add on prediction and planning.

> Richer data does not mean better data. It does not mean more actionable data either.

How actionable you make it depends on how good your engineers are. Diverse input data is definitely more actionable than just RGB pixels from images.

> But maybe it's not. Maybe the additional flops required for sensor fusion would be more useful if given over to the planning part of the problem. After all, you don't have infinite cycles.

On the contrary, it reduces your compute requirements. For example, adding lidar means you are getting direct distance measurements instead of wasting compute cycles predicting distance using voxels. It's a much more effective use of compute.

You are getting direct distance measurements for some things, but not for everything, so you still need a system that is predicting distance using voxels.

LIDAR doesn't work great in the rain/snow, so you still need a system that is predicting distance using voxels.

Doesn't really sound like less compute to me. I feel like I'm repeating myself but again this is what I mean by scale matters. A self-driving system that performs better in perfect conditions is not even a good system if it doesn't work in the rain.

Which is the point the detractors of "fuse all the things" have been making. Don't get me wrong, I think probably those same detractors (e.g. Elon Musk) have also not been entirely intellectually honest with their arguments and have ulterior motives like cost savings and saving face, but that doesn't make the arguments wrong.

Indeed. Ultimately humans do it remarkably well with just vision, too, when they’re not intoxicated, distracted, or falling asleep.

Yeah, adding sensors can sometimes make things worse.

Example? There are plenty of examples where mistakes that Tesla's cameras make (being blinded by the sun, having trouble with emergency vehicles) are no problem for systems with more sensors.

This talk was worth watching (any Karpathy or Carmack talk is imo): https://www.youtube.com/watch?v=g6bOwQdCJrc

Specifically this is the relevant part: https://youtu.be/g6bOwQdCJrc?t=1370

They removed radar because it was making things worse.

I interpreted that statement by Karpathy as an accidental admission of Tesla's own limitations and not of the approach itself. (Note: I'm upvoting all your comments because they're very reasonable and supported.)

> A car equipped with more sensors, will always be better at handling more situations

Yes, integrating seamlessly is the hard part.

For Tesla, they found the increased sensor fusion complexity made the overall system less reliable, which was what informed their attempt at vision only. Karpathy went into this on Lex Fridman's podcast.

Only with better sensor fusion algorithms that worst case the overall system performance doesn't degrade with additional sensors. Tesla could have hit compute bounds in order to meet latency requirements for example (or they couldn't find the right technique/algorithms).

Sounds like a hard problem but I imagine this would eventually be overcome in the future. More sensors is definitely the future.

This is retroactive justification for using low-cost sensors. There's a chance they get lucky and pull it off with only cameras, but the chance of Tesla doing that is much less likely than Waymo getting their sooner with more sensors (or even even the Waymo driver achieving it with only vision if they chose. Using a full suite of sensors gives Waymo much better training data if they ever chose to use vision-only.

I think Tesla does use Lidar in training for what it's worth (at least I recall someone telling me this - I think it was a friend that worked there but not 100% sure I'm remembering correctly).

It's possible the lidar approach only really works with up to date high resolution maps and without that you end up in a local max you can't really escape with that approach.

That's been Tesla's argument up to this point anyway and I don't think the success in cities really proves things either way.

It may not matter since robotaxis in cities is still worth a lot and highway driving is mostly a solved problem.

True full self driving though will probably require solving vision - in that case I think the argument that lidar is a local max could very well be true.

> It's possible the lidar approach only really works with up to date high resolution maps and without that you end up in a local max you can't really escape with that approach.

Sorry, but you’re just repeating Elon Musk’s buzzwords like “local maximum”. It doesn’t make sense. There’s no such thing as “lidar approach”. All sensor inputs are fused (early/late fusion) and run through perception algorithms. Lidar is used for localization with maps, but that’s not even its primary use (that’s object detection). And it does not require maps to be up to date either.

I expect Tesla's rationalization will suddenly change when the cost for lidar units drops to hundreds of dollars. At that point, Waymo will have a huge head start.

Lidar cost has already dropped a lot. Waymo’s lidar on their I-Pace is 90% cheaper than the previous gen. We’re also seeing car manufacturers (GM, Volvo/Polestar, Mercedes, Chinese OEMs) include inexpensive lidar units for driver assistance.

Tesla’s problem is that they’ve promised their existing cars on the road are capable of full self driving. They can’t add hardware to them, so they will maintain the “camera is sufficient” stance for a long time.

Birds fly by eating bugs and flapping wings, but that doesn't mean we should build planes that way. Tesla is stubbornly handicapping itself by cheaping-out on sensors because they think they can hail-mary yolo themselves all the way full self driving.

The Tesla's do have more than two cameras despite us having only two eyes. I don't think that analogy totally holds. It's more that because roads (and road signs) are designed for human vision and human drivers in order to handle all cases you really need to solve vision and if you do that sufficiently then the other sensors are redundant anyway.

High resolution mapping works for localized solutions, but not in the general case.

I'm glad both groups are working on different approaches, it'll be better for us in the long run imo.

High resolution maps are only one part of the equation. Everyone is working on a generic perception/prediction/planning stack that works everywhere. Only the map changes between different cities. This is how Waymo/Cruise have been able to deploy in different geographies.

Reply on news.ycombinator.com