Waymo recalls software after two self-driving cars hit the same truck
cnn.com
cnn.com
Autonomous vehicles have various redundant systems built-in that can take priority and override false positives.
I was previously under the assumption that one of the really important reasons for Lidar is that it can get you closer to an absolute truth about whether something is a solid object, and where that hypothetically solid object is relative to the position of the vehicle, regardless of what the classifier thinks it is seeing.
So did the lidar fail to read the solid object, or was the lidar, was it de-prioritized? or was it simply not available as a fallback?
Presumably Radar and proximity sensors were also involved. What were they doing?
This is a fascinating edge case, and I hope to hear about the real reason for the 2 incidents.
The brakes don’t respond immediately - you need to be able to detect that a collision is imminent several seconds before it actually occurs.
This means you have to also successfully exclude all the scenarios where you are very close to another car, but a collision is not imminent because the car will be out of the way by the time you get there.
Yes, at some point before impact the Waymo probably figured out that it was about to collide. But not soon enough to do anything about it.
If the max speed is 35mph, that allows a good braking system to respond by safely stopping from LIDAR info 99% of the time.
People will take them if they're priced right and they can do things in the car without having to pay attention to the road.
I get that self driving software is difficult. But theres no excuse for this type of accident.
But take an even slightly more complex example: you're on a two lane roadway and the car in the other lane changes into your lane, leaving inadequate stopping distance for you. You brake as hard as you safely can (maybe you have a too-close follower, too), but still there will be a few seconds when you could not, in fact, avert a collision if for some reason the car in front braked.
I have no idea what the legal situation would be: is it their fault if the crash happens within 3 seconds but yours if it happens after you've had time but failed to restablish your needed stopping distance?
Honestly even in the simple one lane case, I doubt you can slam your brakes on the interstate for no reason then expect to avoid any liability for the crash, blaming your follower for following too close.
Driving has a bunch of rules, then an awful lot of common sense and social interaction on top of them to make things actually work.
Even if your software is as good as it can be, I doubt you'll be able to get them to recognise how to resolve deadlocks. Which would also involve severe hindrance to emergency vehicles.
The planning software would want to slam on the brakes without predicting that the blob of sensor data in front of you is going to continue moving forward at highway speeds. That motion prediction enables the planning software to know that the space in front of your vehicle will be unoccupied by the time you reach it.
A similar prediction error was the reason Cruise rear ended the bendy bus in SF a while back. It segmented the front and rear halves of the bus as two separate entities rather than a connected one, and mispredicted the motion of the rear half of the bus.
I think we're all on the same page about this part but what's confusing and hilarious is why would the correct answer ever be to drive into an unmoving object?
If they tried to avoid the truck and swerved and hit a different vehicle there would be no confusion here. But the self driving algorithm is effectively committing suicide (Kamikaze). That's novel.
My guess is that the self-driving car was not able to recognize the truck until it was very close and the sudden appearance of the truck is interpreted by the algorithm as if the truck is moving very fast. And the best answer in that case would be to let the truck pass (basically do what the waymo did).
But that means the lidar information about the shape not moving is being deprioritized in favor of the recognized object being calculated to move fast. A situation which could only really occur if a speeding vehicle plowed through a stationary object.
Fantastic solution for avoiding a situation like this -> https://www.youtube.com/watch?v=BbjjlvOxDYk
But a bad solution for avoiding a stationary object.
It’s why it’s so difficult to do (actually) and the ability to do it well is just as much about the risk appetite of the one responsible as anything else - because knowing if a car is likely to pull out at the light into traffic, or how likely someone is to be hiding in a bush or not is really hard. But that is what humans deal with all the time while driving.
Because no one can actually know the future, and predicting the future is fundamentally risky. And knowing when to hold ‘em, and when to fold ‘em is really more of an AGI type thing.
(It's been quite some years since I worked on vision-based self-driving, so my experience is non-zero but also quite dated.)
I understand we have to have explanations or we can't fix them, but it's just as important to understand this should never have happened even WITH the described failure.
If I had to guess, there's code to avoid stopping at every little thing and that code took precedence (otherwise rides would not be enjoyable). And I get the competing interests here but there must be a comparison to humans when these incidents happen.
I would actually put money that this is the cause of most crashes involving multiple moving cars. Hell, a friend of mine got into an accident two weeks ago where they t-boned somebody that turned onto a median when they didn't expect it.
If your friend saw them and never attempted to stop then your friend needs to lose their license.
or what if they're driving a bus and they have to keep above 60 mph!
---
I guess it's my fault, I didn't list every single contingency possible...
This is literally the cause of almost every human accident.
Imagine you're driving. There's a car in front of you, also driving, at the same speed as you. Do you immediately slam on the brakes? No, because you EXPECT them to keep driving. That is how driving works.
If, suddenly, they do something unexpected - like slam on the brakes, that might cause an accident. Because ... they moved in an unexpected way.
I honestly can't even figure out what you meant to say.
https://injured.ca/5-top-causes-of-car-accidents-in-ontario/
> they're going to slam on the brakes and the only way that's [hitting the other vehicle] going to happen is if momentum is too high.
there's a difference between hitting something and driving into something.
Well obviously even Waymo agrees, given that they're recalling vehicles to mitigate the issue.
https://waymo.com/blog/2024/02/voluntary-recall-of-our-previ...
I'd sat through 'five whys' style postmortems before, but it was reading air safety investigation reports that finally got me to understand it and make it a useful part of how we get better at our jobs.
By comparison, the way we're investigating and responding to self-driving safety incidents still seems very primitive. Why is that?
There is an explicit culture and mechanism of blamelessness around safety concerns and minor violations/deviations, which is incredibly helpful. Read about the ASRS* program (admin'd by NASA, with anonymity for non-intentional issues, prohibition on use of submissions for enforcement purposes, and explicit "get out punishment" card from the FAA): https://asrs.arc.nasa.gov/overview/immunity.html
The FAA is also explicitly including "evidence of voluntary compliance" and "just culture" in its approach to aviation safety, and explicitly changed its goal from enforcement and proof to ensuring compliance [with enforcement being only one available tool]: https://www.faa.gov/about/initiatives/cp (PDF pres: https://www.faa.gov/sites/faa.gov/files/2022-09/The%20Compli... )
I'd also read a bunch of aviation reports: https://www.ntsb.gov/Pages/monthly.aspx (more detailed reports are available approximately 2 years after the occurrence date and more details are available for fatal or air carrier occurrences, so if you don't care which ones to read, filter for those to start).
If you're more video oriented, watch @blancolirio, @NTSBgov, @AirSafetyInstitute, or @pilot-debrief. (I'd skip @ProbableCause-DanGryder.)
For a short summary, there is an intense focus on determining the facts (who, what, when, where, maybe some guesses as to why) and drawing conclusions about primary and contributing causes from there.
* Aviation Safety Reporting System
In this case two self-driving cars crashed into another road vehicle because they failed to recognise (in time) which direction it was moving. Waymo should be commended for having voluntarily issued a software recall, but this problem is severe enough that the decision shouldn't really be up to Waymo's good judgement.
One difference with this situation in terms of the public perception/discussion though is that, say in the 1960s, air safety wasn't very good compared to today, but still there was no question of eliminating air travel altogether due to safety issues. Today there is definitely an anti-self-driving contingent that would like to hype up every accident to get the self driving companies shut down entirely.
A simple lidar moving object segmentation, which doesn't even know what it's looking at but can always spit out reasonable path predictions, would probably have saved them.
I think Mobileye is doing something like this, but they release so little data, which is always full of marketing bullshit, that it is hard to know what exactly they are working on.
We're now getting to see where autonomy needs to develop "spider sense": the scene in front of me feels wrong because some element isn't following the expected behavior maybe in ways that can't really be rationalized about, so we'll become much more conservative/defensive when dealing with it.
if prediction != reality
{mismatch+= whatever the time tick is}
if mismatch > second (or two)
{abort driver assist}Each model can potentially predict longer into the future but also has more complexity and things that can go wrong. So you keep track of how well each model is doing (on an object basis) and if one level is failing then you fall back on a stupider one. You might also want to increase caution if your models are not doing well (lower speed and increased safety distance).
But if everything is fine, everything is fine, everything is fine, and then all hell breaks lose? We are not as good at dealing with that.
We also know how to hold individuals accountable for independent accidents. We know we won't get justice when people will inevitably start to get killed by standard corporate greed, incompetence, enshittification.
All, no. Enough to make a difference, easy.
Black Friday in February! In-Store offer only!
$99 Playstation 5 to first 100 customers at each Walmart location!
When I make a dumb mistake, the other drivers rarely learn from it.
At least that is how I (as a non-expert) imagine these models work -- the model has an excellent chance of crashing at every new unique situation it encounters out of a nearly unlimited set of possible situations (which implies a high frequency of encountering new situations).
This would be common for a debt recovery or when a city impounded the vehicle where it's taken without cooperation of the owner.
https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2F...
> People worry that ways and times [self-driving cars] are unsafe (separate from overall rates) will be unusual, less-predictable, or involve a novel risk-profile.
In this example, having a secretly cursed vehicle configuration is something that we don't normally think of as a risk-factor from human drivers.
_______
As an exaggerated thought experiment, imagine that autonomous driving achieves a miraculous reduction in overall accident/injury rate down to just 10% of when humans were in charge... However of the accidents that still happen, half are spooky events where every car on the road targets the same victim for no discernible reason.
From the perspective of short-term utilitarianism, an unqualified success, but it's easy to see why it would be a cause of concern that could block adoption.