How many sensors for autonomous driving?
semiengineering.com
semiengineering.com
It is not an open question and I hate it when writers frame it this way. Camera only (specifically, monocular camera only) systems literally cannot be safer than ones with sensor fusion right now. This may change in the future at some point, but it's not a question right now it is a fact.
Setting aside comparisons to humans for a second (will get back to this), monocular cameras can only provide relative depth. You can guess the absolute depth with your neural net but the estimates are pretty garbage. Unfortunately, robots can't work/plan with this input. The way any typical robotics stack works is that it relies on an absolute/measured understanding of the world in order to make its plans.
That isn't to say that one day with sufficiently powerful ML and better representations we would be totally unable to use mono (relative) depth. People argue that humans don't really use our stereoscopic depth past ~10m or so and that's a fair point. But we also don't plan the way robots do. We don't require accurate measurements of distance and size. When you're squeezing your car into a parking spot you don't measure your car and then measure the spot to know if it'll fit. You just know. You just do it. And it's a guesstimate (so sometimes humans make mistakes and we hit stuff). Robots don't work this way (for now), so their sensors cannot work this way either (for now).
From how humans drive, its pretty clear that there exists some latent space representation of immediate surroundings inside our brains that doesn't require a lot of data. If you had a driving sim wheel and 4 monitors for each direction + 3 smaller ones for rear view mirror, connected to a real world car with sufficiently high definition cameras, you could probably drive the car remotely as well as you could in real life, all because the images would map to the same latent space.
But the advantage that humans have is that we have an innate understanding of basic physics from experience in interacting with the world, which we can deduce from something simple as a 2d representation, and that is very much a big part of that latent space. You wouldn't be able to drive a car if you didn't have some "understanding" of things like velocity, acceleration, object collision, e.t.c
So my bet is that just like with LLMs, there will be research published at some point that given certain frames in a video, it will be able to extrapolate the physical interactions that will occur, including things like collision, relative distances, and so on. Once that is in place, self driving systems will get MASSIVELY better.
Taking things to the extreme, perhaps it’s actually a networking problem.
Cars should have the ability to send signals and hear signals from the cars around it.
Imagine if Car A could improve its own understanding of the environment using inputs/sensor data from nearby Car B.
We already know for sure that we can’t solve (1), which means we have to solve (2). Therefore, car-to-car communication is, at best, a value add, not the enabling technology.
The most basic example of how this could be helpful is if the car ahead of you turns a sharp corner and crashes into a truck stopped in the road. Without car-to-car networking, you won't brake until the crash is in your line of sight.
Have you ever seen those youtube videos of massive car pile ups on highways caused by a crash, and then a cascade of additional crashes afterwards? E.g. icy conditions or dense fog. What if the original crash could communicate to cars behind it, wouldn't that be helpful if the crash isn't yet in the driver's (or car's) line of sight?
I agree "not much would change" overnight. It's just another input for the car's software to have at its disposal.
With the current hardware on the roads, I don't think it's technically possible for autos to achieve legitimate self-driving (if that's even the goal anymore?) - there are way too many edge cases that are way too difficult to solve for with just software.
Even if every car is required to be part of the network, there may be badly maintained cars that don’t work properly, or even malicious cars, that send wrong data on purpose.
You can't rely on this in real time because urban canyons make it hard to get consistent cell signal (for one thing), but you can definitely improve your models on this data once the data's been uploaded to your offline systems, and some SDC companies do this.
At this point it's all about edge cases. Certain edge cases are impossible to overcome with just software + cameras alone.
It involves a situation with networked self driving cars.
I disagree. When in a car, we are using more than our eyes. We have sound as well, of course, something that we provide feedback even in the quietest cars. We also have the ability to feel vibration, gravity and acceleration. Sitting in a sim without at least some of these additional forms of feedback would be a different skill.
There was another even where they took another sim driver and put him in a real drift car, and he was able to drift very well.
Both vids are on youtube. Yes, real world driving has more variables, and yes, the racing drivers had force feedback wheels, but in general, if a person is able to control a car so well as to put the virtual wheel in the right square foot of the virtual track to take a corner optimally, its probably likely that most people could drive very well solely from visual feedback. Sound and IMUs can provide additional correctional information, but the key point still remains, is that whatever software runs has to deduce physics from visual images.
Self driving cars obviously have microphones and accelerometers too.
I recommend watching this NOVA video on human perception. When doing any number of task, especially ones we do commonly we're using a ton of unconscious perception and prediction based upon our internal representation of physics and human modeling.
For example when I was younger I was noticing that I was commonly aware that a car was going to get over before it did so. I kept an eye out trying to determine why this was the case and I noticed two things. One is people commonly turn their head and check the mirrors before they even signal to get over. The other is they'll make a slight jerk of the wheel in the direction before making the lane change.
Self-driving is still a robotics problem, and robots are probablistic operators with many component dependencies. If you have 3 99% reliable systems strung together running 24 hours a day, that's 43 minutes a day that it will be unreliable ((1 - .99^3)*1440). Multi-modality allows your systems to provide redundancy for one another and reduce the accumulating correlated errors.
You could drive fine at 30fps on a regular monitor (SDR). More fps would help with aggressive/sporty driving of course.
You’d also lose depth perception entirely, which can’t be good for your driving.
What? This is preposterous.
Have you tried playing a shooter video game at 30 FPS? It's atrocious, you get rekt. There is a reason all gamers are getting 120 FPS and up.
30 FPS means 33 ms of latency. Driving on a highway, car moves over a meter before the camera even detects an obstacle. The display has it's own input lag, so does the operating system. Your total latency is going to be over 100ms, so the car will have travelled several meters. If a motorcyclist in front of you falls, you will feel the car crashing into his body before the image even appears on the screen.
Also, if you truly are only a few meters behind a motorcyclist when driving at highway speeds, by definition you are being unsafe. The rule I learned in driving school was roughly 1 car length per 10mph of space, so you should be ~90 feet (~30 meters) away.
Finally, the average reaction time for people driving in real life is something like 3/4 of a second. 750ms to transition from accelerating to braking. A self-driving car being able to make decisions in the 100ms time frame is FAR superior.
Eh, kind of...
https://youtu.be/HU6LfXNeQM4?t=1987
Check out this NOVA video on how limited your acute vision actually is. It is only by rapidly moving our eyes around that we have high quality vision. In the places you are not looking your brain is computing what it thinks is happening, not actually watching it.
Whether it's fine to be less safe in certain situations because it's safer overall is a different question.
The existence of peripheral vision disputes that pretty definitively, though.
Tangential: event cameras work more like our eyes but aren't ready for AVs yet.
That being said, this idea of a latent space representation of the world is the right tree to be barking up (imo). The problem with "scale it like an LLM" right now is that 3D scene understanding (currently) requires labels. And LLMs scale the way they do because they don't require labels. They structure the problem as next token prediction and can scale up unsupervised (their state space/vocabulary is also much smaller). And without going into too much detail, myself (and others I know in this field) are actively doing research to resolve these issues so perhaps we really will get there someday.
Until then however. Sensors are king, and anyone selling you "self-driving" without them is lying to you :)
We're at least a decade away from it... (and yes, I've seen the current batch of FSD videos).
However Waymo, Cruise and others do exist. If you haven't already, check out JJRicks videos on YouTube. I think you might be changing the number of years in your estimation ;)
https://media.mbusa.com/releases/mercedes-benz-worlds-first-...
Even if parity with human vision can be attained, humans kill 42,000 other American humans each year on the roads. If human driven cars were invented today, and pitched as killing only 42,000 people per year, the inventor would get thrown into a special prison for supervillains.
> monocular cameras can only provide relative depth
While the environment awareness is nowhere near as good as two or more cameras would be, if you consider the output over time, you get valuable information about the change rate of the environment, i.e. how fast that big thing is getting bigger, which may indicate one should actuate the brakes.
Of course, I'm with the crowd that answers the question with a "how many can we have?" question. The more, the merrier. And the more types, the better - give me polarized light and lidar, sonar, radar, thermal, and whatever else that can be plugged in the car's brain to make it better aware of what happens (and correctly guess what's going to happen) outside it.
Binocular cameras provide absolute depth information, and are an order of magnitude cheaper sensors than the other options.
Since this technology is clearly computationally limited, you should subtract the budget for the sensors from the budget for the computation.
According to the article, the non-camera sensors are in the $1000’s per car range, so the question becomes whether a camera system with an extra $2000 of custom asic / gpu / tpu compute is safer than a computationally-lighter system with a higher bandwidth sensor feed.
I’m guessing camera systems will be the safest economically-viable option, at least until the compute price drops to under a few hundred dollars.
So, assuming multi-camera setups really are first to market, the question then is whether the exotic sensors will ever be able to justify their cost (vs the safety win from adding more cameras and making the computer smarter).
They measured out to 18m & point out that the typical measured limits of angular resolution of the human eye mean that we could extract stereo image information out to 200m or more.
This paper claims to demonstrate stereopsis out to 250m, which is roughly the limit you’d expect from typical human visual acuity: “Stereoscopic perception of real depths at large distances” https://jov.arvojournals.org/article.aspx?articleid=2191614
This paper suggests that steropsis occurs out to somewhere between 20m & 65m before other cues dominate 3D depth perception: “The Role of Binocular Cues in Human Pilot Landing Control” https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.30...
It seems that the claim that stereo vision only occurs in the near field case is probably wrong? Human stereo vision is much more capable than that & if it reaches out to significantly > 20m is surely being used when driving?
As seen on their website [1], and confirmed numerous times, they have monocular vision around the car and, though having three front-facing cameras, they each have different focal lengths and are located next to each other and thus can not operate as binocular vision.
Human life is worth about $10 million (in US), that's a bit more than the sensor does. If one in 10,000 of camera-only car causes deaths, then it's not economically viable.
A London bus costs about $300,000, it is economically viable. Why is $1,000 sensor a problem. It is definitely viable to be installed on busses and trucks. Maybe you need to get out of the mindset of personal cars. It is not a viable business model, and it is not a viable model of dealing with congestion either.
The $100,000 system would only ever make sense if the fleet was already entirely self driving, and money for other life saving stuff like the environment and health care also hit suitable diminishing returns. Of course, by then, the cheap systems will have improved.
This argument holds for any non-negative dollar value you place on human life.
It is also independent of who owns the vehicles. Money the bus fleet spends in expensive self driving pulls money away from bus stop upgrades, pollution controls, etc, etc.
Don't forget some interesting scenarios like dust particles or mud on one of the cameras (or windshield if cameras are located behind the windshield) or rain beading and distorting the image thus breaking the feature matcher and resulting disparity estimates.
Next, to "see" further, a stereo rig needs to have a decent baseline. For example, in a classic KITTI dataset, the baseline is approximately 0.54m which is much larger than, for example, human eyes (0.065m). Such baseline, 54cm, together with focal length, which, if I remember correctly, is about 720px in case of KITTI vehicle cameras, would give about 388m in the ideal case of being able to detect 1 pixel disparity. But detecting 1px of D is very difficult in practice - don't forget you will be running your algo on a car with limited compute resources. Say, you can have around 5px of D, that means max depth of around 77m - comparable to older Velodyne LiDARs.
Some of the issues I mentioned are not specific to stereovision (e.g. you need to calibrate monocular cameras as well and so on), just wanted to point out that stereovision does not magically enable depth perception. The solution would likely be a combination of monocular and stereo cameras, combined with SfM (Structure from Motion) and depth-from-stereo algorithms.
“Monocular” cameras > no “absolute” depth > less safe
The last leap is not well justified.
Also, cars with vision based driving have multiple cameras. Whats the difference between a “binocular” camera and two “monocular” cameras?
How does a “binocular” camera get better depth information?
Is using multiple cameras to drive sensor fusion?
Why is absolute depth a strict safety win? How do you know how the sensor details translate to the final safety of the full system?
If this is just a handwavey upper bound on safety, how do you know that such a system can’t be safe enough for its design goals?
If humans with only one eye are able to drive, why wouldn’t mono surround vision be at least as good as that?
> Whats the difference between a “binocular” camera and two “monocular” cameras?
For the camera itself, nothing. They are probably referring to the implementation. You can have two cameras side by side but unless you are using homography to estimate depth from the two images, then your setup is monocular.
> How does a “binocular” camera get better depth information?
A pixel in two images (with known separation) will have a geometric relationship that can be used to extract depth information. This is a lot faster than alternative methods with a single camera and multiple images.
> Is using multiple cameras to drive sensor fusion?
This is really just a question of semantics.
> Why is absolute depth a strict safety win?
Why is it better to have two eyes than one? You can be more certain about what you are seeing.
> If this is just a handwavey upper bound on safety, how do you know that such a system can’t be safe enough for its design goals?
If you had a system with infinite compute you could probably do enough math to calculate absolute depth with 100% certainty. I believe you can already extract absolute depth with something called bundle adjustment-- but it requires multiple images since you are relying on parallax effects. It is also computationally expensive.
> If humans with only one eye are able to drive, why wouldn’t mono surround vision be at least as good as that?
Computers are not humans.
If you can get reliable depth information, the algorithm needed to avoid hitting stationary and slow-moving objects is extremely simple.
Is the stationary object in our path, of nontrivial size, and about to enter our minimum stopping distance? If yes, do we have a swerve planned that will let us safely avoid it? If no, emergency stop.
Because this logic is simple and well defined you can audit the implementation to the high standards applied to things like aircraft autopilot systems.
And it'll work even if the stationary object is something that didn't appear in your training data - you know the algorithm will work the same even if that concrete barrier is painted with some cheery flowers, or if that fire truck is airport yellow instead of the normal red.
Of course, this relies on the assumption you can get reliable depth information. If your depth sensor gets confused by a cloud of dust while driving in the desert, or gets blinded by the light of the setting sun, or is unable to detect a barbed wire fence, things are no longer quite so simple....
> If this is just a handwavey upper bound on safety, how do you know that such a system can’t be safe enough for its design goals?
Personally I would say that in freeway driving, a self-driving car should be able to avoid 100% of collisions with clearly visible stationary objects in dry, well lit conditions when all system components are in normal working condition.
But I agree that current software (Tesla?) is not able to do that in the same way. So it may need more sensors until the software gets better.
In theory cameras should also be able to see more than humans. They can have a wider angle, higher contrast, higher resolution and better low-light vision than the human eye.
That said, I think what you're referring to is more extreme perspectives that shift in ways the car cannot drive and you are correct that this would aid in reconstruction. This is how NERF models do their 3D reconstruction (https://nerfies.github.io/).
Can't cameras do this by just comparing frame 1 to frame 2?
Stereoscopic vision in humans only works for nearby objects. The divergence for far away objects is not sufficient for this. You may think you can tell something is 50 or 55 meters away through stereoscopic vision, but you can't. That's your brain estimating based on effectively a single image.
That said reality is not a single image, it's a moving image, a video. Monocular video can still be used to estimate object distance in motion.
Eventually AI will be good enough to work better than humans with just a camera. The problem is we're not there yet, and what Tesla is doing is irresponsible. They should've added LIDAR and used that to train their camera-only models, until they're ready to take over.
I'm not sure what kind of systems you're referring to with "monocular cameras", but if you look at the visualization in a Tesla with FSD Beta, it's actually really good at detecting the position of everything. And that's with pretty bad cameras and not a lot of compute.
Only rarely you'll see Tesla's FSD mess up because of perception, the vast majority of times they mess up is just the software being dumb with planning.
It's pretty easy to see, even as someone with very little experience, the benefits of stereo vision over monocular. In addition to the depth stuff it's a lot easier/faster to create your point clouds from disparity maps.
The second paper I reference in this comment https://news.ycombinator.com/edit?id=36232198 claims that humans can maintain stereopsis out to 250m.
That’s a huge difference from 10m & if true suggests that human drivers might well use 3D vision when driving.
IMO it's better for us to focus on what the robots can and cannot do right now, and focus on solving those problems :)
Thanks for the sources though, those papers are definitely neat and I'll be taking a look when I get a chance.
How exactly does a certain type of sensor help with this? Isn’t the problem entirely based on a software model of the world?
... and I want to grab the guy who says that by the collar and scream in their face "The whole point is to build something that can do better than a human."
My argument on autonomous driving is that it can't just match the safety of the average human driver. It needs to be 100-1000x better, unquestionably better for everyone, always. Until that happens it's a dead end. I think that's probably only achievable with LIDAR, which will come down in price as volumes ramp up.
The thing that surprises me is that Tesla has invested so much into various hardware technology (even their own silicon), but completely ignored LIDAR. With their resources, volume and investment, they could have reduced the cost to a tenth of what it is now. They go for big challenges, but decided not to try on that one.
Probably very expensive.
But wasn't there some rumor already some time ago, that apple wanted to make a car?
But cars are a different ballpark, than consumer electronics. So they are likely cautious to not burn their money there.
Again, I'm not arguing that Apple might not make a number of other design decisions that could lead to the car costing much more than that. I'm just saying that the cost of the Vision Pro sensors won't be the reason.
And to be clear: I don't think the Vision Pro implies that Apple would needlessly increase the price of a hypothetical car. The Vision Pro is as expensive as it is not because of some perceived "Apple Tax" but because Apple had a vision (ahem) of what they wanted to build, and it was at the very limits of technical possibility, and they chose to release it now (2024) instead of five years from now. For comparison, the original iPad, which was anticipated to cost a thousand dollars, shipped at half that price.
Ah. And then there is Tesla ditching an external temperature sensor and relying on weather info for the GPS location to save a few bucks. I think the automotive industry works on a completely different standard when talking about parts and margins.
and I'd want to see if cars+roads+whole infrastructure designed from scratch by Apple
/s
Best at what?
I'm sure it does, but that's also because you need much greater degrees of precision and accuracy to anchor virtual content in the real world. Inches and centimeters matter here, whereas an autonomous vehicle can get away with coarser measurement.
And it costs about 9% of what a new Model 3 would run you. It's also coming to market almost 5 years after the Tesla HW3 stack that runs almost all cars on the roads to which you're making the comparison.
> My argument on autonomous driving is that it can't just match the safety of the average human driver. It needs to be 100-1000x better, unquestionably better for everyone, always.
That sounds like maybe a marketing argument? You're saying that if it was only 2x better at some visceral metric (total traffic deaths, say) that you think people wouldn't accept it and it would be regulated out of existence, either by liability concerns or actual lawmaking?
The counter argument is that those metrics are, after all, visceral. The US has 40k traffic fatalities per year. You really think the government is (or courts are) going to kill a technology that could save 20k people every year?
No. "Merely better" is the standard that will stick.
> I think that's probably only achievable with LIDAR
Everyone on your side of the argument does, but frankly this has become an argument of faith and not substance. LIDAR capability has become a proxy for "not Tesla", so people line up along their brand loyalties. For myself, LIDAR is actually the technology looking like it isn't panning out. It's not getting cheaper or more reliable. LIDAR autonomy isn't getting better nearly as fast as vision solutions are iterating.
Most importantly, LIDAR at its best still only gives you the 3D shape of the world, and that's not enough. LIDAR doesn't help with lane selection, it doesn't help with traffic behavior modelling, it doesn't help with sign reading. It doesn't help with merge strategies. All that stuff has to be done with vision anyway.
The "don't hit stuff" bar has, quite frankly, already been crossed. Tesla's don't hit stuff either. If you give them LIDAR, they.... still won't hit stuff. But all the work, even for LIDAR vehicles, is on the camera side.
You don't spend much time on the Tesla subreddits. The only reason there aren't more crashes is because most people seem to be reasonably good at keeping their hands on the wheel to take over. Every day people post videos of the wacky choices FSD makes. The most interesting part is that frequently the graphics on the screen suggest it can actually detect the correct lane markings and such, and still chooses to cut the corners and cross into other lanes.
Anybody who's being honest and has watched FSD from the beginning is under no illusions that real self-driving Teslas are happening anytime soon, or with the current hardware. It's not even close.
No, but I watch my car drive me around every day. This always devolves into an argument like this. I wanted to talk about LIDAR and limitations, not flame about collisions that woulda/maybe/coulda happened in funny internet videos.
Believe what you want. But if they aren't crashing they aren't crashing. And you won't get anyone to ban cars that don't crash.
In some situations with well marked, simple road design, it works great. Give it a multiple-lane turn intersection to navigate and things get interesting. Give it a three lane turn where the outermost lane is fairly sharp, and it will just cut across the middle lane because it seems to have a built-in limit on how hard it can turn the wheel.
What's noteworthy is that it doesn't significantly improve with each revision. It'll get a little better, then regress, over and over, but still the fundamentals are unreliable. At some point maybe it becomes clear that the current strategy is topped out.
I do think it tends to be better behaved than AP on the highway, though, so I look forward to a stack merge there.
Except actual accident statistics, you mean.
> Give it a multiple-lane turn intersection to navigate and things get interesting. Give it a three lane turn where the outermost lane is fairly sharp, and it will just cut across the middle lane because it seems to have a built-in limit on how hard it can turn the wheel.
Now you've digressed into navigation choices and not safety. And... I agree! This is exactly what I was talking about upthread. It's an issue for real improvement, and notably not something that can be helped by LIDAR at all. Tesla is actually way ahead here, as I see it. Waymo and Cruise operate in heavily constrained environments that disallow challenging maneuvers entirely.
(FWIW: that three-lane turn behavior you're talking about doesn't seem familiar. I don't see that with my car, and we have a few intersecions like that around here. In general multi-lane turns are marked, and the car is quite good about figuring it out.)
I thought we had gotten past that statistic? AP, in particular, gets used in what is already the safest driving situation there is. Those statistics can only be compared to human drivers in exactly that same narrow circumstance. If we were talking about FSD, then we'd need to see statistics where the human never touches the wheel; saving the car from it's own mistakes is cheating ;).
> Now you've digressed into navigation choices and not safety.
I think I understand what you are saying but I'm not sure I agree -- incorrectly navigating across the lane lines, especially when the car indicates it sees them, is definitely a safety problem in my view.
I expect that it's kinda like the silly wipers. If you live somewhere with the right kind of rain, you think "what is everyone complaining about?" but if you live in the PNW where we get a lot of misty rain, you end up cursing at the car regularly during the winter months. Some places have just the right roads for FSD, and those people think it's pretty great. But then other people get constant failures and wonder why the heck they spent 6, 12, or 15 grand on something that doesn't deliver.
I'm glad someone has a good experience with it, even when it doesn't match my own. But I also didn't buy it outright, so I just cancel the subscription after trying it again to see how things stand.
> It does always devolve into the same discussion, I agree. All of the evidence says it doesn't work well. One guy says "but it does for me!"
ajross:
It works on my machine!
Yours isn't crashing (yet), but that doesn't mean they aren't crashing. https://www.tesladeaths.com/ shows a lot of crashes. Last year most crashes involving automated driving systems were shown to be teslas (https://www.theguardian.com/us-news/2022/jun/15/tesla-us-car...)
Worse, both Tesla and the NHTSA have been caught covering up safety problems too so it's even harder to say what's actually happening on the roads.
In fact it's so bad that you just have to assume their data is wildly incomplete because the not at fault fatality rate for a car just being in traffic (i.e. Tesla drivers killed because someone else hits them) isn't correctly represented.
Yes, some of those links involve teslas being hit by others, and many likely involve cases where the driver is totally at fault. Most of the links are just reports of crashes still under investigation.(this one for example can't even provide the time when the accident took place https://web.archive.org/web/20230304195927/https://www.bigco...)
Still, that's okay, because this isn't a scientific paper, it's just a website collecting data on accidents involving teslas. Each one would still have to be investigated to find out what the cause of the crash was before you could draw much from the raw numbers, but what we can tell unambiguously is that these vehicles do in fact crash. Here's a case where autopilot killed someone just a few months ago (https://abc7news.com/tesla-autopilot-crash-driver-assist-cra...)
It'll be framed as a technology that will kill 20k people per year.
Let's say self-driving does reduce the number of deaths by half. Then there are different classes of people who ride in cars:
1. People who would have died without self-driving, and did in fact die with self-driving. 2. People who would have died without self-driving, and didn't because of self-driving.
If you leave it at that, you might think that of the 40K number you mentioned, with self-driving being 2x as safe we'd have 20K in class 1, and 20K in class 2. But that's not accurate. There is an additional class of people:
3. People who wouldn't have died without self-driving, and who died because of some failure of self-driving.
Since drivers are supposed to be ready to take control at any time, car manufacturers want public perception to be that there is no category 3 -- and with good reason, because:
The problem self-driving car manufacturers have is that people in category 2 are anonymous -- almost invisible. At the end of the year we can look at the totals and say, "yup, there are 20,000 people out there alive because of self-driving." But if the numbers are actually:
1. 19,000 2. 20,000 3. 1,000
Then people aren't going to be thinking about the 20,000 anonymous lives saved by self driving. They're going to be thinking of the 1,000 very obvious and named people in category 3, who would still be alive today were it not for the fact that their car drove them into a semi.
I don't know what ratio of category 2 to 3 is necessary for people to accept self-driving as a benefit, but I'm betting that 2x as safe isn't going to get category 3 low enough to make the case.
Category 3 deaths need the be astronomically low for people to widely except fully autonomous driving.
This is just plain wrong. Lidar is getting cheaper by the year. They are now so cheap you’re starting see them in consumer vehicles from Volvo, Mercedes, Cadillac, Nio, XPeng, BYD, etc. Self driving companies have also slashed lidar costs drastically. Waymo, for example, had a 90% cost reduction for their 5th gen Lidar.
This is the opposite of a technology “not panning out”. It’s looking like Tesla is the outlier here because they made a bad bet.
> LIDAR autonomy isn't getting better nearly as fast as vision solutions are iterating.
Lidar autonomy is the only one proven to work without a driver, so this is a very strange take. We’re seeing robotaxi companies starting to expand driverless operations to big cities. Vision solutions are the only ones that still do not have a single driverless mile.
It seems like all your arguments fit into what you’re accusing others of — that you’re the one using lidar as a proxy for “not Tesla” and getting basic facts wrong.
I'm googling, and not finding a reference here. As far as I can tell not one of these manufacturers (or anyone else) is shipping a consumer vehicle with a LIDAR sensor. You're just citing press releases about plans, I guess?
Look, the argument goes like this:
1. LIDAR gives somewhat better depth info, and slightly inferior vision.
2. Depth info from camera devices is proven sufficient. Again, Teslas don't hit stuff due to sensor failures, period (lots of nitpicking in this thread over that framing, here I'm being a bit more precise).
3. All the remaining hard problems are about recognition and decisionmaking, all of which gets sourced from vision data, not a point cloud.
The conclusion being that LIDAR isn't worth it. It's not giving significant advantages. If you were designing an autonomy system from scratch today, you wouldn't bet it on LIDAR (especially as there's no proven off the shelf device!).
XPeng P5, XPeng G9, Nio ET7 are all shipping with lidar. You'll also see them in many other models starting this year: Mercedes S-class/EQS w/ Drive Pilot, Volvo EX90, Volvo XC90, Audi A6L/A7L/A8L/Q8, Polestar 3, etc. The list is growing [1].
> 2. Depth info from camera devices is proven sufficient. Again, Teslas don't hit stuff due to sensor failures, period (lots of nitpicking in this thread over that framing, here I'm being a bit more precise).
Teslas absolutely hit stuff. Plenty of examples in Reddit and YouTube about FSD hitting curbs, bollards and other things. We also don't know any accident numbers about Tesla because they actively hide it. They skirt CA DMV rules by not reporting it and have a very questionable methodology for what they consider an accident (they don't count it if airbags don't deploy).
> 3. All the remaining hard problems are about recognition and decisionmaking, all of which gets sourced from vision data, not a point cloud.
This is wrong, too. Object detection and decision making uses fused input from cameras, lidar and radar. They don't just use cameras. There's plenty of literature to look up on sensor fusion and how they're being used throughout the perception/behavior prediction stack.
> The conclusion being that LIDAR isn't worth it. It's not giving significant advantages.
Only if you lack the ability to do sensor fusion or if you have already falsely promised customers their cars are capable of fully autonomous driving and painted yourself into a corner. Only one company stands out here.
[1] https://cnevpost.com/2023/04/21/shanghai-auto-show-nearly-40...
1. It's not really a contest. Lidar is measurably better.
2. Error bars in perception tend to accumulate throughout the stack and manifest as serious errors like bad velocity estimates for stationary or slow moving vehicles.
3. LIDAR helps recognition too. Planning and pathing are both helped by LIDAR.
Honestly, there aren't many other sensors on an AV with the same bang-for-the-buck as LIDAR. It's easily up there with cameras for "why would you even consider not having it?" for a modern driverless vehicle, which is why everyone operating driverless vehicles uses them (Waymo, Cruise, Mobileye, Baidu, etc).
Lane lines and road signs in particular typically use reflective paints that are very easy to detect in LiDAR, but beyond that, you can approximate material composition of a LiDAR scene pretty easily.
Another big point you’re missing is that LiDAR can provide control points to photometric sensor fusion systems. While there are also purely photo based control point matching systems, they’re much more complex and require nontrivial offline preprocessing.
Have you heard of this miracle technology that is like 100x safer than cars, and kills zero people a year in most countries, and we've had it for 50 years? called high speed rail?
Covid is still killing more than 20K people a year, and no-one gives a damn. We have this miracle technology called ventilation, it reduces you chance of infection like 4x. You can buy air purifiers for $200 at Ikea and they work. You can install Upper-Room Ultraviolet Germicidal Irradiation, CDC has a whole page about them.
If saving 20K people would be the top priority, Nuclear power would replace coal in the 70's, air purifiers would be legally required in every office and coming into work with Covid would be a criminal offense.
> LIDAR autonomy isn't getting better nearly as fast as vision solutions are iterating.
Why do you believe this to be the case - is there a measurement or estimate I can see of how fast visual solutions are iterating?
IIRC a number of cars, even Subaru, have better cameras than a Tesla. Really, the cameras Tesla included on the Model 3/Y are fairly mediocre off-the-shelf hardware from years ago. But they do have more of them pointed in different directions, so there's that.
Are you saying this from datasheets, or looking at the camera images?
Tesla cameras aren't optimized for humans, their Bayer matrix is likely designed for day and night machine vision including infrared, so the colors might look washed out to you. I heard at one point they were using RCCC and later moved to RCCB [1]. If I'm not mistaken C allows all light, including infrared. Cameras designed for human viewing are RGGB.
You want a high-resolution LIDAR. The trouble is, they cost too much. That's due to the low volume, not the technology. A few years back, there were several LIDAR startups using various approaches. There were the rotating machinery guys, the flash LIDAR guys, the MEMS mirror guys, and some exotics. I was expecting the flash LIDAR guys to win out, because that approach has no moving parts. Continental, the big auto parts company, acquired Advanced Scientific Concepts, a LIDAR startup, and demoed a nice little unit. But nobody wanted enough of them to justify setting up full scale manufacturing. ASC sells some space-qualified units, and the Dragon spacecraft uses them for docking. Works fine, costs too much.
Another of the startups, Luminar, made its founder a billionaire without the company having shipped much product. "Luminar has not generated positive cash flows from operating activities and has an accumulated deficit of $1.3 billion as of December 31, 2022." - Wikipedia. They've announced many partnerships and have been at this since 2012, but you still can't order a LIDAR from their web site.
Velodyne makes those spinning things Waymo uses. They had the first automotive LIDAR, a big car-top spinning wheel, at the DARPA Grand Challenge. It fell off the vehicle. The technology has improved since, but it's still expensive. Velodyne just merged with Ouster, which sold similar spinners. Ouster's web site has a "Order Now" button, but it leads to an email onboarding form, not a price list.
Many of these things need custom indium gallium arsenide detector chips, which would be far cheaper if the market was 100,000 a month instead of 100 a month. This technology hasn't scaled up in volume yet. That's one thing holding this back.
If you want coverage over a full circle, you either need one of those silly looking domes on top of the car roof, or multiple units, each with a narrower field of vision. Waymo has both. Works fine, costs too much.
On the radar side, there's been improvement in both processing and resolution. Not as much as expected over 20 years. Automotive radar has moved up from 24 GhZ to 77 GhZ, so resolution can now potentially be sub-centimeter. True millimeter radar (300GhZ and up) has been demoed [1] but not deployed beyond lab systems. Once you get up there, it's almost as good as LIDAR. Humans show up in detail. That's going to be a useful technology once it gets into commercial products.
So the sensor situation is pretty good, but expensive because it can't mooch off some other high volume technology such as cell phones.
[1] https://www.fhr.fraunhofer.de/en/the-institute/core-competen...
What I'm getting at is, there's no reason to assume that the current amount of information presented on a roadway (stripes, lights, stopsigns, etc.) is sufficient for autonomous driving. Or maybe it is, but adding a few more inexpensive things would make it that much easier to achieve.
Hence, either /you/ drive, or /the car/ drives. If the car doesn't understand what is going on, it should stop. (Humans really suck at monitoring boring, monotonous processes. We cannot expect people to do so in a self-driving car for hour after hour on the off chance something happens.)
I don't mean so people can navigate; I mean so people can complain to a city or district when a road is improperly marked.
Most roads aren't even properly painted ._.
After all does anyone care if the plane they are in is flapping its wings?
Firetruck >_____________< Fire Hydrant
Something that looks flat but is actually a fire hose means you should stop instead of driving over it - this was actually issue in Phoenix(?)
A homeless lady was killed by a self-driving car. She was walking a loaded bicycle, at 2am, on a road that doesn't usually have pedestrians. The self-driving car couldn't decide if she was a bike or a human so it didn't brake and the human-driver backup had zoned out as the car had made 99.99% of the correct decisions.
A self-driving car is like a jigsaw puzzle - there's one way to do it right, and infinite ways for it to be completely or subtly wrong.
"What do you see" and "What does it mean"
Humans and cameras have the same "What do you see" input, but the "What does it mean" category hasn't been conquered by cars.
It's great that we have cars that can warn us when there's something in our blind spot, something behind us while we back up, or when we drift out of our lane. Until the software can perform as well as we can, any more than that is a liability.
That isn't quite right. There can be situations where the human is better, so long as in the majority of situations the machine is better. This is why I said it is a statistical question. I'm not demanding perfection, I'm asking for better than humans overall. Of course if the machine knows it is in a situation where it isn't as good it is fine to make the human drive.
You use way more than two vision sensors in your head. But first, lets talk about those vision sensors. They are mounted on an articulated scanning platform with numerous degrees of freedom. The dynamic range is astounding, and that dynamic range can be easily augmented with shaded lenses that can be mounted and removed as needed via an extremely reliable pick-and-place mechanism. The vision sensors are really equipped with dual-sensing technology for day- and night-mode.
But, you also have extremely sensitive accelerometers in your ears. And force-sensing in your feet to measure applied acceleration/braking controls. And force-sensing in your hands to measure applied steering force, and measure road roughness. You also are sitting on a force sensor that serves as a back-up accelerometer. You have audio sensors that provide feedback about road surface, vehicle performance, and other agents sharing the road.
Yeah, that kind of agent prediction is a biggie. Agent prediction is something journalists don't talk about much, but is a huge research topic for AV practitioners.
And since you make note of expected behavior of objects... my all-time favorite bug report out of Waymo (that made the public news) is a classic classifier/predictor/planner corner case leading to deadlock. Classifier identified a cyclist and labeled as "stopped" if cyclist has foot on the ground, and "moving" if both feet on pedals. Predictor would plot a trajectory for moving cyclist, planner decides action based on predicted agent behavior. So... Waymo vehicle and cyclist on two corners of 4-way stop. Cyclist is stopped, doing a perfectly balanced track stand with both feet clipped in. Feet on pedals so cyclist is labeled as moving agent... planner yields right-of-way to cyclist according to rules-of-the-road. Deadlock -- nobody moves.
She was homeless so the shape of her bike wasn't a proper bike so it got confused.
Very different circumstances. I don’t think this was a classifier defect. Not that I was ever a fan of Uber’s development program, but in this case I can’t fault the platform.
Most automobiles don't run on legs and airplanes are not flapping their wings either.
If Tesla (or any self driving player, but the Tesla FSD stack drives a million miles/day across the US right now) published a road quality score, home buyers would quickly start paying attention and then home owners would quickly start paying attention. This would push forward the political will to fix the roads, at least in places where low road quality is driven by budgeting problems rather than outright poverty.
True, but the problem is that they're often not. That said, I'd rather see infrastructure spending on maintaining the roads, paint, and signage, than adding features specifically for autonomous driving (that will also likely be unmaintained and become unreliable over time).
I think most people would argue that the taxes they're paying now should cover it and if that's not enough we could try things like diverting tax money from waste, eliminating earmarks, or maybe even start forcing corporations and billionaires to pay their fair share. People want nice roads and are willing to pay taxes for it, but the burden shouldn't fall entirely on the shrinking middle class.
But the vast majority of the difficulty in autonomous driving is simply dealing with the messiness of the real world. For example, many of the most serious and notable autonomous driving crashes have been the result of faulty road conditions (e.g. https://www.theverge.com/2020/2/25/21153320/tesla-autopilot-... - in that case, the driver wasn't paying attention, but the fact that lane markers were worn down is what caused Autopilot to lose lane tracking, and the fact that crash attenuator in front of the barrier was damaged and not replaced contributed to the driver's death).
Government can't even keep decent lane lines on the roads, and thus I think an autonomous driving system that had to depend on magnets or radios (that failed if those were out) would be even more of a non-starter.
If you had this, driving period would be safer, automated or not. And that's before getting into inclement weather conditions where, say, lines or signage may not be visible.
(https://www.teslarati.com/tesla-hardware-4-hd-radar-first-lo...)
Example uses 64 kWh if constantly used.
https://www.proxyparts.com/car-parts-stock/result/part-numbe...
https://ev-database.org/car/1423/Hyundai-Kona-Electric-64-kW...
Indeed, the parts are for the Hyundai Kona, with either the 39 and 64 kWh batteries.
If the examples _really_ used 64kW, that would be the equivalent to the average consumption of 51 US houses.
It’s 750kW!!!!
Nor is “professional commercial driver” a job that could be performed by a Neanderthal or a 6 year old student driver of today despite nearly identical hardware and software. I would never ride across Manhattan every day in a car driven by the latter two humanoids and I doubt most anyone would.
Ideally these systems should only be certified after performing demonstrably better than a teenage student driver given a series of the most complex, adversarial driving workloads in deteriorating conditions with disabled and malfunctioning sensors.
On the other hand, today’s billionaires may actually be close to achieving fully autonomous and acceptably failsafe non-human driving on today’s roads, and they might also be exploring Mars within my lifetime. Personally I wouldn’t bet my life on it.
Self driving is a difficult problem because you need to:
- Decipher the nature of the objects in the 3d space you are moving on.
- Predict how their physical relationship to your current position will change according to current timeline and intentions of these objects.
- Explore the plane of possible deviations for each of these objects and be able to react accordingly.
I think this is an oversimplification. People don't ask that question because humans have many many more senses they drive with than just our eyes. Humans even have accelerometers. If someone said "I'm deaf, blind in one eye, and can't sense acceleration" people would probably ask if they are fit to drive. Just being deaf is enough to start that conversation. The AARP even mentions that if someone is deaf, they should seriously consider not driving. The conversation of sensor fusion is important because humans require sensor fusion to be effective drivers too.
https://www.aarp.org/auto/driver-safety/driving-with-hearing...
This reminds me of all the conversations about AI And LLMs now - people are making false comparisons because they don’t have a grasp of basic cognitive science, things we’ve known for 50 or more years
It really is astonishing gow much goes into it.
I am terrible at the sims. I am literally seconds faster in a real car (on a 62 second circuit) than the sim, due to all the additional sensory input in reality. I can barely land a simulated plane but have not messed up a landing since training over three decades ago.
So, yes, it is astonishing how much all our senses go into driving, beyond sight.
Or using google maps in a language you can’t read!
Coming from Europe with the ECE standard DOT standards suck. Much more glare.
And then the lifted cars & trucks with no adjustments.
And it's especially bad in places that don't have some yearly car check.
When I first moved here it felt like 'awesome, no need to get in line for my car check'.
Now I'm like: yea, it helps. Implement those car checks.
Not always. If you don't spot that your engine temperature is rising, or your fuel level is low, you might suddenly find yourself sitting in a non-running car on a busy expressway, as opposed to having a least a few minutes to find a safe place to pull over.
Best to develop a habit of continually scanning the gauges, mirrors, while your primary focus is on the road ahead. That way you are aware of the operating condition of your car and the traffic all around you, not just in front of you. You only need quick glances at the dashboard and mirrors to stay on top of things.
What I meant to say, is that having 2 eyes or 20 eyes will not help if you are not able to decipher scenarios that require a semantic understanding of the current real life movie you are in.
A cyclist made sign with the hand too brief to catch, or a soccer ball just run to street so maybe a child is running after it, or a temporary deviation on the road where the police expects you to drive off road for a few seconds and is sending you manual signs.
None of the current self driving solutions is able to handle these. 1 or 20 cameras, 2 Lidars or 50 Lidars will do nothing to address the problem.
The smarter problem would be changing the controls/UX for cars to be more like a hoverboard or balance/attention and gesture based than twisting a wheel around, abstracting that to remote telemetry, and then work on using ML to replacing that remote driver.
Autonomous vehicle companies have picked the unsolvable moonshot problem instead of improving the controls UX of driving to where it provides greater leverage to human ability. Make the driving experience more like a motorcycle, which is an extension of your body, instead of a mediated service where driving is reduced to a transaction. Self driving cars is the perfect example of sprawling corporate groupthink, where someone has a poorly formed idea and everyone aligns around managing a response to it instead of asking whether it's the right problem to solve. For example, I think the Apple version of a car wouldn't be self driving, it would be one that feels like it's controlled by thought.
Maybe that's Musk's Neuralink play, and self driving was a distraction to set all auto industry competitors on a goose chase. If he's not that strategic, it could at least provide an out, where a gesture controlled Tesla would outperform.
Not just flying, but (emergency) autolanding is now available:
* https://en.wikipedia.org/wiki/Autoland#Emergency_autoland
* https://skiesmag.com/features/virtual-co-pilot-garmin-autola...
I'm sorry, what? The difficulty of self-driving cars is not about controlling the actual vehicle in space, it's about the other dynamic agents involved and getting an accurate view of the physical world.
The last line of your comment is just pure nonsense. Tesla's failed self-driving was a 5d chess move by Musk? Come on.
Sometimes simple is better.
Who the hell would want this? My guess is that 99% of the people that would want this already do drive motorcycles, exactly for that reason.
But for the vast majority of the rest of this, I don't want driving to "provide greater leverage to human ability". I just want it to get me from point A to point B, ideally without me thinking about it so I can do other things.
Go ahead and do it then. Show them how it is done. :)
> Self driving cars is the perfect example of sprawling corporate groupthink, where someone has a poorly formed idea and everyone aligns around managing a response to it instead of asking whether it's the right problem to solve.
Or maybe people thought about the problem for more than hot second and come to a different conclusion than you do.
Your comment makes it clear that you are thinking about the joy of driving. You are talking about motorbikes, and UX, and being one with the car. I'm happy for you for that, and I hope you will find a lot of joy doing that. That on the other hand is not the problem self-driving cars are out to change.
Every single thing in a modern world have traveled a ton. Transportation of raw materials, transportation of parts, transportation of the finished product. Then on top of that people can't make up their mind if they want to be in A or in B and seems to change location all the time. This is economically significant. Driving for joy, is a cool hobby, but is a tiny sliver of the whole economy.
People and companies working on self driving cars believe that the time has come when we have enough compute that we can automate the act of driving and be at least as safe as vehicle driven by a human. If they are right, and these autonomous systems can be cheaper than human drivers then those who can make such machines can capture a portion of all the wages of every activity served by a commercial driver today.
> mediated service where driving is reduced to a transaction.
Do you think CEO of Walmart cares about the user experience of driving when they hire a small army to drive supply trucks to supply their stores? It is already reduced to a transaction. Uber is extremely successful. Do you think people care about a car becoming an extension of their body when they hail one?
Currently people who drive because they enjoy it, and people who drive because they have to are all jumbled into one mass. If we have commercially viable self driving systems you will see who is driving for the joy of it, and who is driving to get stuff or themselves from A to B. I suspect we will see that it will become just an other hobby. Like going skiing or horse riding. Many will enjoy it, there will be probably a whole cottage industry catering to those who would like to "become one with the machine" and etc.
One of my weekend projects is a POC demo for how something like this works. Maybe I'll launch it here, or maybe I'll use it to establish a new autonomous region with an air defense perimeter, tbh, I haven't quite decided, but your insight is valuable.
I'm saying what people think is a "self-driving car" is effectively a fully autonomous robot that has to navigate among humans, and presenting it as just solving a car operation problem is misleading.
The problem they are solving is creating a virtual android to replace taxi, bus and truck drivers, which is surely a great and noble pursuit, but as a consumer product, it's the wrong problem.
Some of the smartest people I have ever met have given it a lot of thought and disagree with me, but when you look at what every single other generalized robot product has succeeded at, it's in scaling human ability and in very limited tasks. Self-driving cars do neither of these things. It's not a car problem and using cars is a misleading constraint.
Your maximum upside is some nuanced variation of a train or a streetcar on dedicated roads or paths/rails that is totally dependant on new public infrastructure to support it. Maybe you get self-driving golf carts and ATVs for limited contexts, just like other robots, but a general purpose one to replace drivers without dedicated lanes, air routes, or infrastructure is implausible.
I'd focus on solving autonomous navigation in more narrow contexts like dedicated lanes and rails, flying, trails, and anything that doesn't have to deal with human obstacles. Otherwise, the failure mode of self driving cars is just a hunter/killer robot with limited commercial applications.
Nice! Please share it if you can. Sounds very exciting.
> Maybe I'll launch it here, or maybe I'll use it to establish a new autonomous region with an air defense perimeter
I say keep your options open and do both. ;)
That's not true: what there is, is 3.2 trillion vehicle-miles driven annually (in the US) vs 12.5 billion aircraft-miles flown -- a factor of 400 difference.
Imagine the reduction of accident rates if, at any moment, there were 1/400 of the vehicles on the road, and every vehicle trip started with a checklist inspection by a pilot who had passed a physical exam in the last year?
Not to mention a slew of other requirements like 'Don't operate the vehicle with insufficient sleep', 'Don't operate the vehicle if you had even a drop of alcohol in the last 8/12 hours', etc..
https://techcrunch.com/2020/05/22/scale-ai-releases-free-lid...
I imagine 99% of the collected data is proprietary and hidden from public view, so it becomes a question of taking corporate claims at face value, which has never been that great of an idea.
Case in point: it's hard to imagine that electric semis hauling goods hundreds of miles can do so safely without human operators involved at some stages on the route. Hence, even with decent autonomous systems having a (paid!) driver in the truck at all times is the rationale thing to do - and yes, that means labor costs in that sector won't be changed much by self-driving AI technology. It will likely increase safety, as the trucks can drive themselves autonomously on the open freeway more safely than a (tired) human driver can.
Sure LIDAR etc is expensive, but so maybe not that one, but in general I would have thought that many cheap & cheerful sensors is the way to go especially if you're going to be feeding it into a neural net anyway that is well suited to unscrambling that signal spaghetti
[1] https://www.theautopian.com/the-advanced-driving-assist-syst...
If you were to truly count every sensor connected to every chip, there'd be hundreds or thousands of sensors. Only some of them are used. Different vehicles in a testing fleet will have slightly different numbers as well, due to the various component changes that are in-flight at any given time.
So yes, you absolutely do need an IMU for self-driving.
Funny. Just today: https://imgur.io/gallery/hMpaVzC
This is useful for detecting stability problems like over- or understeer.
If the steering is pointing slightly left, the car is moving rapidly right, and both rear wheels are turning at roughly the same speed and way faster than the fronts, you've lost traction and are oversteering like crazy.