In the end I think self-driving regulations will require depth checking beyond computer vision as it can be tricked on any new situation. Depth checks using LiDAR are extremely efficient up to a football field away down to the direction someone is facing. RADAR is not as good but better than video/flat 2D depth detection though is limited by range, however it does work in weather where LiDAR doesn't and computer vision struggles.
Autopilot and now FSD on Teslas doesn't have depth checking beyond visual/cameras. They removed the RADAR/sonar and have zero physical world depth checking currently. Tesla recently instead of adding LiDAR, they [just removed RADAR to rely on computer vision alone even more](https://www.cnbc.com/2021/05/25/tesla-ditching-radar-for-aut...).
Self-driving cars need cameras and physical depth checking sensors like LiDAR, or at least RADAR. Telsa has only cameras and some sensors but not for depth anymore, that is insane.
Humans have essentially LiDAR like quick depth testing. Humans have hearing for RADAR like input. For autonomous units, depth may be actually MORE important than vision in many scenarios. Humans have inherent depth checking with 3D space, movement, sound, lighting, feel, atmosphere, air, pressure, situational awareness, etc that computer vision converted to 2D flat images for depth checking will never be able to replicate.
A human can glance at a scene and know how far things are not just by vision but by how that vision changes with these distance inputs. Humans are able to detect 3D/2D imagery easily where it is all based on 2D with a camera. LiDAR is faster than humans at depth checking in the actual physical world not just from an image flattened.
With just cameras, no LiDAR OR RADAR, depth can be fooled.
Like this: [TESLA KEEPS “SLAMMING ON THE BRAKES” WHEN IT SEES STOP SIGN ON BILLBOARD](https://futurism.com/the-byte/tesla-slamming-brakes-sees-sto...)
Or like this: There is the [yellow light, Tesla thinking a Moon is a yellow light because Telsas have zero depth checking equipment now that they removed RADAR and refuse to integrate LiDAR](https://interestingengineering.com/moon-tricks-teslas-full-s...).
LIDAR or humans have instant depth processing and can easily tell the sign is far away, cameras alone cannot.
LiDAR and humans can sense changes in motion, cameras cannot and even RADAR struggles with dimension (frame to frame changes).
LiDAR is better than humans on changes in motion, depth, seeing all around always, and much faster at all those things.
[LiDAR vs. RADAR](https://www.fierceelectronics.com/components/lidar-vs-radar)
> Most autonomous vehicle manufacturers including Google, Uber, and Toyota rely heavily on the LiDAR systems to navigate the vehicle. The LiDAR sensors are often used to generate detailed maps of the immediate surroundings such as pedestrians, speed breakers, dividers, and other vehicles. Its ability to create a three-dimensional image is one of the reasons why most automakers are keenly interested in developing this technology with the sole exception of the famous automaker Tesla. Tesla's self-driving cars rely on RADAR technology as the primary sensor.
> High-end LiDAR sensors can identify the details of a few centimeters at more than 100 meters. For example, Waymo's LiDAR system not only detects pedestrians but it can also tell which direction they’re facing. Thus, the autonomous vehicle can accurately predict where the pedestrian will walk. The high-level of accuracy also allows it to see details such as a cyclist waving to let you pass, two football fields away while driving at full speed with incredible accuracy. Waymo has also managed to cut the price of LiDAR sensors by almost 90% in the recent years. A single unit with a price tag of $75,000 a few years ago will now cost just $7,500, making this technology affordable.
> However, this technology also comes with a few distinct disadvantages. The LiDAR system can readily detect objects located in the range of 30 meters to 200 meters. But, when it comes to identifying objects in the vicinity, the system is a big letdown. It works well in all light conditions, but the performance starts to dwindle in the snow, fog, rain, and dusty weather conditions. It also provides a poor optical recognition. That’s why, self-driving car manufacturers such as Google often use LIDAR along with secondary sensors such as cameras and ultrasonic sensors.
> The RADAR system, on the other hand, is relatively less expensive. Cost is one of the reasons why Tesla has chosen this technology over LiDAR. It also works equally well in all weather conditions such as fog, rain, and snow, and dust. However, it is less angularly accurate than LiDAR as it loses the sight of the target vehicle on curves. It may get confused if multiple objects are placed very close to each other. For example, it may consider two small cars in the vicinity as one large vehicle and send wrong proximity signal. Unlike the LiDAR system, RADAR can determine relative traffic speed or the velocity of a moving object accurately using the Doppler frequency shift.
> Though Tesla has been heavily criticized for using RADAR as the primary sensor, it has managed to improve the processing capabilities of its primary sensor allowing it to see through heavy rain, fog, dust, and even a car in front of it. However, besides the primary RADAR sensor, the new Tesla vehicles will also have 8 cameras, 12 ultrasonic sensors, and the new onboard computing system. In other words, both technologies work best when used in combination with cameras and ultrasonic sensors.
LiDAR and depth detection will be needed, no matter how good the pure computer vision solutions get.
The accidents with Teslas were the Autopilot running into large trucks with white trailers that blended with the sky so it just rammed into it thinking it was all sky. LiDAR would have been able to tell distance and dimension which would have solved those issues.
[Even the most recent crash where the Tesla hit an overturned truck would have been not a problem with LiDAR](https://www.latimes.com/california/story/2021-05-16/tesla-dr...). If you ask me sonar, radar and cameras are not enough, just cameras is dangerous.
Eventually I think either Tesla will have to have all these or regulations will require LiDAR in addition to other tools like sonar/radar if desired and cameras/sensors of all current types and more. LiDAR when it is cheaper will get more points almost like Kinect and each iteration of that will be safer and more like how humans see. The point cloud tools on iPhone Pro/Max are a good example of how nice it is.
Human distance detection is closer to LiDAR than RADAR. We can easily tell when something is far in the distance and to worry or not about it. We can easily detect the sky from a diesel trailer even when they are the same colors. That is the problem with RADAR only, it can be confused by those things due to detail and dimension especially on turns like the stop sign one is. We don't shoot out RADAR or lasers to check distance but we innately understand distance with just a glance not just from vision alone though.
Humans can be tricked by distance but as we move the dimension and distance becomes more clear, that is exactly LiDARs best feature and RADARs/CV trouble spot, it isn't as good on turning or moving distance detection. LiDAR was built for that, that is why point clouds are easy to make with it as you move around. LiDAR and humans learn more as they move around or look around. RADAR can actually be a bit confused by that. LiDAR also has more resolution far away, it can see more detail far beyond human vision.
I think in the end on self-driving cars we'll see BOTH LiDAR and RADAR but at least LiDAR in addition to computer vision, they both have pros and cons but LiDAR is by far better at quick distance checks for items further out. This stop sign would be no issue for LiDAR. It really just became economical in terms of using it so it will come down in price and I predict eventually Tesla will also have to use LiDAR in addition.
[Here's an example of where RADAR/cameras were jumpy and caused an accident around the Tesla](https://youtu.be/BnbJvUwbewc?t=262), it safely avoids it but causes traffic around to react and results in an accident. The Tesla changed lanes and then hit the brakes, the car behind was expecting it to keep going, then crash.... dangerous. With LiDAR this would not have been as blocky detection, it would be more precise and not such a dramatic slow down.
Until Tesla has LiDAR it will continue to be confused with things like this: [TESLA AUTOPILOT MISTAKES MOON FOR YELLOW TRAFFIC LIGHT](https://futurism.com/the-byte/tesla-autopilot-mistakes-moon-...) and this: [WATCH TESLA’S FULL SELF-DRIVING MODE STEER TOWARD ONCOMING HIGHWAY TRAFFIC](https://futurism.com/the-byte/watch-tesla-self-driving-steer...). [They are gonna want to fix FSD wanting to drive toward moving trains](https://twitter.com/TaylorOgan/status/1487080178010542085).
In the end I bet future self-driving, when it is level 6, will have computer vision, LiDAR, RADAR and potentially more (data/maps/etc) to help navigate. [Tesla FSD has been adding more of data/maps in which is basically what they said they didn't need to do](https://twitter.com/WholeMarsBlog/status/1488428565347528707), so LiDAR will have to come along eventually. Proof of them [using maps data](https://twitter.com/IdiocracySpace/status/148843350997893939...) and possibly previous driver data.
LiDAR is extremely fast and accurate up to multiple football fields even determining which way a pedestrian is facing, with high fidelity. That is much better than a camera or computer vision. LiDAR can get many reads in the time you could do one complete CV pass.
The best process is to do depth checks across the viewable area, then overly the results of the computer vision tick, then again check the differences and when it comes to depth, go with the LiDAR feedback as video can be wrong/tricked. This is probably how the cars that use LiDAR are doing it (Waymo/Cruise/etc), the equipment is also on the roof for better vision/coverage.
Tesla is going to be massively behind when the regulations come down that self driving needs depth checks beyond CV. You can still base lots of the process on CV. However doing depth solely in CV can be tricked and uses massive amounts of processing power which ends up with faster usage of power. CV will always be limited in distance, dimension and depth to LiDAR and physical world detection/sensors.
To think a Tesla can drive you without intervention or watching it closely, there will eventually be a distance confusion and it won't go well. The name Autopilot was better as it inferred like a plane where you still have to watch it, though planes are much further apart. The name Full Self Driving should be changed immediately even in beta, it is going to be ripe for lawsuits and problems.
Tesla is trying to brute force self-driving and it will have some scary edge cases, always.