Tesla’s CV Approach to Autonomous Driving Built an Unassailable Lead in FSD

Tesla fanboys seem to frequently forget the following facts:

* People using lidar also use deep learning.

* Lidar and HD maps are totally orthogonal concepts with nothing to do with each other. Lidar helps you avoid running into trucks without HD maps. Camera-based methods can use HD maps too.

* Lidars are less affected by rain and snow than cameras are (thanks to larger optical aperture, multiple returns, and faster "shutter speed" due to nanosecond pulses rather than 1/30 s exposures).

* Tesla's "big data" is not more effective than a concerted active learning data collection campaign with a moderate-sized fleet.

* HD maps are very cheap to create.

* Lidars may in fact be cheaper than the GPU you're going to need to run your monocular depth network, while being more accurate and more robust.

I could not get through the entire article with straight face. It's riddled with fallacies and misconceptions on the problems currently facing self driving. It was especially this horrible take that put the credibility of this article into question:

> "A simple analogy can illustrate the conceptual difference between Computer Vision and LiDAR. Imagine two students, where one is just cramming and memorizing the content (LiDAR), while the other one is trying to really understand the material and truly learn it (Tesla FSD). The student that learned the material (Tesla FSD) will be able to answer the exam questions correctly, even if the questions on the exam are swapped, the questions are rephrased, or new components are added to the questions, while the student that memorized the content (LiDAR) will likely fail the exam."

Tesla's advantage isn't even it's computer vision systems - computer vision models don't exactly scale like language transformers (where larger, sparser parameters make better models). There hasn't been any significant advancements in vision models since Faster-RCNNs or YOLO, which is close to five years old now. Especially if you want to compare Tesla's SotA against Waymo, which has an army of Captcha labelers and a large plethora of example images.

The goldmine is in identifying and navigating around rare edge cases - data that can only be obtained with hundreds of thousands of hours of real world driving. It has very little to do with the correct set of camera and calibration configurations. The novel research is deep into safety verification strategies. Like how do can we predict human or object behavior by using miliseconds of prior movement of an object? Or how to use human arm movements and eye pupils to determine if a pedestrian or other driver is distracted? Can we avoid an accident if we have a better behavioral understanding of the scene?

What lead? The main advantage of Tesla is that they have the guts to put beta quality software in cars and get away with it (from a regulatory perspective) up to now.

As can be witnessed in loads of videos on YouTube, the current FSD betas still pretty much require full attention (I would say even more attention than driving youself because you have to watch the environment AND the behaviour of Autopilot). And these demos are on American roads which are among the easiest to drive in the world (generally very wide, lots of traffic lights, large intersections, etc.).

Bad weather conditons (which are a thing outside of southern California) are even more problematic. So while the tech might or might not be more advanced than what other car companies have in their labs, reliably its still pretty much an improved cruise control and in that regard not that far ahead of the competition (all premium brands have that in one way or the other).

Given the early stage, I find the idea of anyone having an "unassailable lead" to be laughable.

The idea that the one with such a lead is Tesla is even more laughable, and I say that as someone that actually likes some significant elements of Tesla's approach. Although I pretty strongly dislike some aspects too.

There seems to be gigantic non-sequitur here where good performance of convolutional neural networks for image recognition is seen as evidence that visual sensors suffice for full driving autonomy and that LIDAR-based approaches are heading in the wrong direction.

There doesn't seem any evidence for the proposition that "if only we can train the model with more data it'll suddenly be good enough". There's an extremely long tail of scenarios, operating in different weather conditions, in different environments, at different times of day, and the driving problem is adaptive - the network needs to predict how others will respond to its own behavior.

The authors dismiss the idea that sensor fusion may provide more additive capability than train-with-m0ar-data but don't actually seem to provide any basis for that.

Another dumb post by someone who believes Tesla operates in a vacuum... Tesla is a very small player in terms of cars produced. Volkswagen has started outselling Tesla in some European countries. And dont forget about GM and Toyota. No lead is unaissalable in a highly competitive market like car manufacturing. And its not like there's no competition from the tech side either. Microsoft, Apple and Google are all working on making self driving cars. Any combination of said tech companies with a large car manufacturer would be a serious threat.

Has the author done any serious work in computer vision? Imagenet is one thing, but there are many many problems that need to be solved before a vision-based vehicle can run reliably. I worked at a self-driving company for a while, and they struggled with issues like multiple target tracking and reasoning about occluded vehicles.

And in a sense, vision is the easy part. There are whole areas like prediction and motion planning where the current state of the art isn't really up to scratch.

hmm, I think the author missed that Waymo drove in snow 3 years ago [0], obviously outside of Pheonix. They also don't seem to understand it's CV + Lidar, not versus... I'd be surprised at a CV only system that can handle snow like seen in the Waymo example.

The rest of the article does not hold up once you realize the author is in an either-or mindset and thinks Waymo is not using CV, which they are, and has vertical vision stack, with arguably better experience at scaling ML

If you want to see actual FSD (without a human in the driver seat), this Waymo beta user has been video documenting on a regular basis: https://www.youtube.com/playlist?list=PL-13jt3ZPb7X6qJTo_MEn...

Having seen the "self-driving" displays from both, Waymo's shows Waymo information, which gives me greater trust in the system. By contrast, there are Tesla videos with some scary moments.

[0] https://www.engadget.com/2018-05-08-waymo-snow-navigation.ht...

"Immediately, the Bolt EUV is, in my experience, the best autonomous driving experience available today."

"But for my money right now, Super Cruise beats Tesla’s Autopilot in real-world usage."

https://electrek.co/2021/03/05/chevy-bolt-super-cruise-autop...

> Why is Computer Vision (using neural networks) superior to LiDAR?

It would be better to compare LiDAR to CMOS sensors. You can apply computer vision techniques to either.

I have spent the last six years working in self-driving and disagree with the article. I am an autonomy engineer at a major self-driving focused company.

First: The choice is between “lidar and cameras” and “cameras alone.” I am not aware of any contenders who have are only using lidar. That means the only downside to using lidar is cost.

Second, the article is incorrect. lidar is extremely reliable for detecting dogs, pedestrians, and anything else you can think of. For lidars with sufficient intensity sensitivity, you can even read the text on signs.

Here’s a list of some tradeoffs for available sensors.

Cost: Sterling Anderson said in a talk at MIT a few years ago “there is no unobtanium in lidar.” Making lidar cheap is a matter of manufacturing scale. Not a matter of new physics. Cameras are still much cheaper and will remain so for some time. This alone might justify choosing cameras for consumer vehicles. The game-changing imaging radars that exist are not cheap.

Long-Tail Events: On a camera-based system without depth sensors, the vehicle must react based on correct identification of obstacles. Consider an image of a pedestrian painted onto the road. A system with depth sensors will not need to stop.

Depth estimation with multiple cameras leaves a lot to be desired. It is bad for untextured objects. Poor illumination conditions will prevent texture from being visible to the cameras. Poor illumination conditions have no effect on lidar/radar.

I would not bet my life on a estimated depth from a monocular camera, no matter how many layers the DNN has.

Weather: Lidar works fine in the rain and snow. Degraded, but fine. Radar works fine in the rain and snow. Cameras can be made to work well, especially if placed in enclosures that self-clean. ATG’s vehicles famously made “whooshing” sounds as their pneumatic lens-cleaners forced water off of their camera lenses.

Time-of-Day: visible light cameras will do poorly. Every system I have seen has degraded camera performance at night. Some systems include an NIR channel to help. You cannot bring enough onboard illumunation to compete with the Sun. Most lidars choose a wavelength that leaves them completely unaffected in day vs night. Ouster has different noise characteristics during the day, but not enough to matter.

Range: At long distances, no commercial sensor can beat the angular resolution of cameras. This is where they shine most. That’s why you see highway-focused systems emphasize cameras so much. Blackmore was a promising path to enabling highway-range lidar capability, but they were bought by Aurora years ago now.

It is possible that cameras are completely sufficient. It is possible that Tesla is even ahead. But this article’s reasons won’t be the causal factor.

The company that builds a functional autonomous car will introduce the largest sea change in transportation since the automobile. The value delivered by each car will be massive. An additional $8,000 for a single lidar is not a dealbreaker. And that’s at today’s costs for a nice Hesai. In 2025 it will be yet smaller. In 2016, the only good lidar on the market was the Velodyne HDL64, which cost $80k. Pucks failed too often.

I don't know how this will pan out, but I've seen a lot of criticism of Tesla "only" using vision, as if it's a ridiculous concept that will never work.

But humans drive... only using vision.

Maybe it's not possible for Tesla to get true FSD using only video data given current technology, but the idea that it's laughable doesn't make any sense to me. People drive in new environments, using only their eyes, all the time.

We will need better-than-vision to drive safely - whiteness all the humans that pile up in multi-car crashes when it gets foggy or snowing.

Lidar, or something like it, will have to be part of the equation given that the best visual processing computers of all time (human brains) don’t get enough data from their systems to make good choices in bad weather.