Notes on Cruise's Pedestrian Accident
danluu.com
danluu.com
> 10-02, 9:29pm: accident occurs
> 10-03, 1:30am: AV back at Cruise facility
I don't think the police would let an ordinary driver have their car back 4 hours after they ran over a pedestrian! I think they'd keep it for some time to aid their investigation. Is Cruise treated differently here, or am I just wrong?
I'm also surprised they seem to be sharing video by starting a video call and then screen-sharing while it plays?? I'm struggling to come up with a better strategy for degrading the quality.
It's a very effective way to prevent the video from being spread through the entire Internet faster than anyone can reasonably act.
That's at least the way aeronautic and European railway investigations happen, and for good reasons - you don't want people holding back evidence for fear of repercussions.
You want time to recover data, investigate the physical condition of the aircraft/vehicle, etc.
How thorough of the job can the police do in the dark from 9:30pm-1:30am?
This completely looks like a special snowflake exception that Cruise, given its behavior afterwards, should not have had... and at this point, should not be trusted to have again.
It is important to remember I think that to the police officers this is just a day in their job they are clocking in and out. They are just following their instructions. It might be life changing for someone who got hit but the world just keeps spinning.
What's to stop Cruise or other bad actors to take possession of the car and then clip the brake lines to claim it was actually an OEM vanilla car mechanical fault and not their AV hardware/software?
It's not that outlandish. You already have them doing a whole bunch of lies of omission and smoke screening their video evidence, enough so to trigger a wave of resignation/firings.
And just like that they lost all interest in the situation.
Unrealistic take: The police should have kept the vehicle and demanded the source code be released so that it may be investigated if proper safety mechanisms have been implemented and if any errors in the systems may have caused it to hit the pedestrian. In the meantime the vehicles running on the same hardware and software should be put on probation.
Realistically: Police cannot investigate this and as long as these vehicles are classified more like computers and less as an aircraft these incidents are "glitches".
Presumably that happens because the NTSB want to examine the airplane. In this case there was no equivalent organization to the NTSB run by the Police and nothing about the car that needed to be examined by the Police to establish what had happened.
Car crashes never lead to dusting for fingerprints.
The telemetry would be coming from Cruise, who is not a trustworthy source. This incident is a perfect example of why we shouldn't rely solely on telemetry - whether you believe they were just an innocent victim of bad internet connections or actively trying to downplay the dragging, it's clear they were not a wholly reliable source of information.
Right, but they didn’t know that then.
That's a good point. If you see a pedestrian hit by a car anywhere near your car, you stop, right? If you see a pedestrian walk out into the road against the lights you at least slow down and/or cover the brake pedal because who knows what they're going to do?
I also think that this feedback loop of "Model the world, take action, observe, update model" is a key component of consciousness (whereas we model ourselves!) and key to AGI. I would not be surprised that AGI comes from a self-driving car development model.
The reality is that nearly all sensory input is novel in some way and a large part of the work the human brain does is quietly filtering out a massive amount of irrelevant input in a way that is computationally infeasible with current technology.
Like the fact that a cyclist on the other side of the road is a hazard to you, not because of the cyclist, but because an impatient driver stuck behind them is likely to pull out suddenly round the cyclist, forcing you to brake.
You might stop, yes. But many humans don't. That channel is mild and tends to be relatively work-safe; it doesn't take much searching youtube to find the rest.
It's hard to avoid thinking that maybe humans should never be allowed to drive automobiles.
I'm not sure that's actually true, but even if I posit it... "properly trained, well rested, and sober" does not actually describe humanity. You can revel in your moral superiority to the average driver, or you can make the roads safer.
AVs do not need to be better than the best human drivers. AVs need to be better than the average human driver. The average human driver is shit.
>Wood shows full video, "reportedly with internet connectivity issues from his home computer"
There's one for the back to office lot.
>wheels were moving at different speeds (because one wheel was spinning on pedestrian's leg)
Eurgh.
But in general, this is Cruise learning what traditional car companies already knew - you need to act with integrity. I don't see how senior executives at cruise could reasonably be allowed to continue to work in any safety critical industry after this.
Compared with whom?
Boeing?
And there's also the climate change, and the ~40,000 deaths per year in car accidents in the US.
[1] https://en.wikipedia.org/wiki/General_Motors_streetcar_consp... [2] https://en.wikipedia.org/wiki/Ford_Pinto#Cost%E2%80%93benefi... [3] https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal
You can be notably bad without being the worst.
Toyota just had a similar scandal where they were faking certifications tests.
Those are 2 largest car makers.
Fiat Chrysler was bribing uaw bosses to get favorable terms.
There's no integrity.
Agreed - but also a key mantra in political and crisis communications: the coverup is what kills you. The coverup (or just the appearance of it) is far more damaging than the acute crisis itself.
They just got caught this time.
The acute crisis says "we fucked up", which is survivable in most cases.
The coverup says "there may be a nearly-infinite number of severe fuckups in our product that you do not know about", which is far less survivable.
Fuckups lose trust. Coverups make you fundamentally untrustable. One is far, far worse than the other.
What do you mean? Cruise is owned by GM. It is a traditional car company. And also traditional car companies do sketchy things literally all the time.
> 10-02, 9:29pm: human-driven Nissan Sentra strikes pedestrian in crosswalk of 4-way intersection at 5th & Market in SF
>Pedestrian entered crosswalk against a red light and "Do Not Walk" signal and then paused in Nissan's lane. Police report cited both driver and pedestrian for code violations and concluded that the driver was "most at fault"
>The impact launched the pedestrian into the path of the Crusie AV
>Cruise AV braked but still hit pedestrian
>After coming to a complete stop, the AV moved to find a safe place to stop, known as a "'minimal risk condition' pullover maneuver (pullover maneuver) or 'secondary movement.'"
>AV drove up to 7.7mph for 20 feet, dragging pedestrian with it
>Nissan driver fled the scene (hit and run)
That seems like quite an astonishing claim.
Likewise ethically, morally or responsibility wise. The "everyone at fault" sometimes happen, but it is not clear at all whether it is majority of cases.
"Contributing to the crash / making the accident worse than it could have been" may be more accurate but longer winded.
But even "almost always everyone is contributing to the crash / making the accident worse than it could have been" is pretty tall claim and would require a lot of evidence. Unless you count "leaving the house in the morning instead of staying home" or something like that as a contribution toward accident.
Legal determinations come after our accepted rules are applied to the situation. Those rules ebb and flow dependent on the nature of the inquiry and the legal standard applied. Even seemingly irrelevant factors such as insurance coverage can be pivotal. Where both divers are covered by the same insurer, there is a higher likelihood that both will be found somewhat at fault. An insurer on the hook for both sides isn't going to spend money arguing forcefully for an all-or-nothing determination. But such things are totally irrelevant to anyone seeking to prevent accidents from happening in the first place.
> Where both divers are covered by the same insurer, there is a higher likelihood that both will be found somewhat at fault.
Not true at all.
And in particular, beyond being fined for not having mandatory insurance, you insurance does not influences at all who goes to jail or who will be fined and for what.
> The legal debate is about who has more or less blame, who has enough to be adjudicated responsible for which resulting injuries.
The legal debate fairly often ends with one driver being ruled as the cause of the accident with no blame spread.
That being said, accidents where someone died or ended up with injuries are large part of what criminal system deals with.
Thought experiment - vehicle A disregards a stop sign and pulls out directly in front of vehicle B* and there is a collision.
I don't see how vehicle B "could have prevented or mitigated a situation" other than by not being on that road at that moment. The driver of vehicle A is completely at fault, B is blameless.
* spoiler, it's actually not a thought experiment: my mother was in Vehicle B when this happened to her when she was coming to pick me up from school when I was about 10 years old. Having your mother arrive at your school in the back of a police car was a unique experience....
- the intersection designer - maintenance workers that didn't clear visibility obstacles - vehicle safety features, both active and passive
et cetera.
Your mother wasn't "to blame", but could she have prevented the situation by driving more defensively? Likely.
I did mess up by not creeping slightly slower but the driver who hit me had far better visibility (I checked from that perspective). I am almost certain they were going too fast for the roadway and may not have been fully paying attention. I didn't just blast through. I was creeping out at very low speed. An attentive driver would have seen me and taken evasive action and/or honked.
The third party that had illegally parked obstructing my view can and drove off while we were sorting out the wreckage. In my shock, it took weeks before I realized that they had also contributed significantly but I had no proof since they'd just driven away.
I was considered fully liable for the crash since I had entered the roadway without the right of way. I accept that I made a small mistake. I understand why the insurance people made that determination. But it still burned to be considered fully at fault considering all the other mistakes (including bad road design) that contributed.
I will bet though that driver B in my crash felt that I was completely at fault and might have described it exactly as your mother did her accident. In his case, I think he likely would have been leaving out his role in not paying attention and driving too fast. There can be more nuance even to seemingly cut and dried cases.
All complex systems are in “failure mode” of some sort at all times. But complex systems that work have enough wiggle room and safety mechanisms that true failure doesn’t happen until there’s some criticality of overlapping errors.
Yes, there are some failure modes that are so devastating that everyone else could have been behaving perfectly and it would have made no difference, but likely many if not most traffic accidents are the confluence of multiple failures, even if most are minor and would never cause an accident on their own.
> 10:30am: Virtual meeting with NHTSA. Wood shows full video, "again having internet connectivity issues causing video to freeze and/or black-out in key places including after initial impact" and again not bringing up or discussing pullover or dragging
Convenient “internet connectivity issues” indeed. Looks a lot like students willingly corrupting zip files when sending assignments in order to gain time when they couldn't finish…
The reason it could not be done with cars is the drivers. Drivers simply cannot be held to the same standard as commercial pilots or staff.
Remove the driver from the equation and it might just be possible to replicate the process.
That assumes regulation that will require "operators" to provide truthful reporting of accidents.
I hadn’t thought of it, but I think the idea of similar investigations for self-driving car accidents is a really great idea.
Though, one recurrent theme of that blog is that blame and persecution are generally counterproductive to understanding the accident and improving safety.
It is called accountability and if nobody is accountable for anything then there is no actual motivation to get businesses and financial institutions to do anything about it.
What is counterproductive is blaming and persecuting wrong things.
The goal is to get companies to be accountable for being committed to taking this process seriously, providing honest information whenever it is needed and then to honestly implement the recommendations once the process is concluded.
We know that technical bugs happen regardless of good intentions. So we should err on the side of being lenient.
Where we should not be lenient about is simple negligence where all of the information was available and the decision was made regardless. For example, if the manufacturer knew that bolts were lying around and decided to save on the costs by being willingly blind to the fact.
And where real persecution should strictly follow is hiding information and evidence that is obviously needed to improve safety.
There's a good chapter on blame that you should read in this book: https://www.simonandschuster.com/books/There-Are-No-Accident...
They could; it would be devastating to the economy, though. I'm all for it though. Too many people get injured and die by auto every day.
It should be unacceptable to design roads or cars in a way that increases the likelihood of injury or death and every serious injury or death should be investigated to determine if the road or car contributed to the accident.
I wonder whether this flagged alarms in Cruise's systems when the pedestrian crossed in a potentially dangerous situation. These kind of 2nd and 3rd order conditions seems particularly hard to train for.
Those are all things an alert human driver should be ready and able to do. I would expect a quality AV system to do them better than a human, but in this case, none of those things appeared to happen. This AV decided “pedestrian is out of my lane, continue accelerating!” Only stopping when the pedestrian was right in front of the vehicle.
That's what human drivers do too. If you slowed down anticipating that a pedestrian's collision with some other car might fling their body in front of your car, you'd never make it to your destination. There's just too many crazy people in SF
- AV starts moving at -9.2s, after light changes
- Prediction output shows pedestrian path crossing AV travel lane: -7.7s
- Pedestrian leaves AV's travel lane: -5.3s
- Pedestrian pauses at crosswalk : -4.7s
- Contact b/w Nissan and Pedestrian; -2.7sAlso, based on how the Cruise AVs we saw in Houston perform, this was all but inevitable. They drove extremely erratically and casually drove down wrong way lanes several times.
What? Who ran risk management for this company? If you're facing program halt you keep running the plan until you run out of money to pay your employees, and then keep running it after if you can get volunteers. If these people actually cared about AVs and safety, or even the continued existence of their company, they would work to extract every single detail that could possibly help any AV or robotics/AI company in the future, up to a very high cost for themselves. Deeply unserious.
Yes, my suggested solution applies to everyone. I suggested it to you because you have fear, many people do not.
The solutions certainly lower individual risk.
It also sounded ironic to me, I thought others might notice and appreciate the irony in my suggestion.
I don't understand how "Vertrauensgrundsatz" applies here.
However, the advanced, LLM based systems with tons of world knowledge (and therefore can easily tell if a pedestrian is abnormal) take tons of processing power and latency, so cannot be deployed in the current AI driving systems. So it'll take 5-10 years for this to be solved.
Also hope you know that Germany isn't the center of the world. Its not even the center in terms of cars anymore. German regulators don't get to dictate the progress of self driving.
You are not afraid of corrupt bureaucrats, but of idealistic innovators trying to change the world?
By the way, there is no way to tame safety via checklists or 'policies'. Boeing sure had checklists and layers of policies trying to control risk, can't save them when they outsource everything to workers who don't give a crap about their job.
But people in the AI world care, they absolutely do care about what they build. Pride and passion in their work beats any checklists, and I'm confident self driving cars will be a massive safety boon for all.
Cruise and Waymo are inviting catastrophe by having their embryonic systems practising on public roads without a human in the driver's seat. None of these systems are anywhere near good enough to be let loose unmonitored. I know it's unpopular to praise Tesla here, but in my view they deserve some credit for sitting pretty at SAE L2 while everyone else seems obsessed with quickly impressing investors with a half-baked rush to L3 or L4.
Sure, but that once can be still pretty far away for general conditions.
And you can't get that data with a safety driver in the car. A safety driver should always override in emergency situations, whether necessary or not.
Also, we could lower amount of people who die to human drivers too and I am all for it. Speed limits, regulations about car design, higher driving age, better public transport, there is a lot that could be done.
Of course, not that long ago, a fair number of people were looking forward to a near-future when they'd have a personal robo-taxi at their beck and call. And it's pretty clear that's not happening anytime soon.
Predictions of the future have always been poor, but ML/AI has thoroughly fucked up any prior intuitions we had around the rate of advancement. So many things progressing more rapidly than we thought possible, while others being an endless slog of local maxima after local maxima.
At this point, even some relative skeptics like Rodney Brooks are probably looking overly optimistic in terms of timelines.
I suspect that current approaches only take us so far and some other things from cognitive science and so forth will need to be pulled in.
Having taken Waymo a number of times in SF now, I have experienced this once. The car stopped and an unsolicited call from customer support came into the car to check on things.
While subjective my experience in Waymo has been positive and feels surprisingly safe.
The car drives conservatively but naturally. It shows a great understanding of other cars and people and cones from LIDAR on its HUD. It takes safe routes and picks safe side streets and alleys for pick up and drop off. There is a real-time support team watching things.
I doubt it’s ready for every city — SF is well equipped for public transportation with lots of well mapped local taxi routes and loading zones in the city.
And there will be surprises and accidents.
But it’s absolutely wild that we’ve reached this level of self driving, and it will be transformative for human kind as it continues to improve.
“Rush” is subjective. Waymo have been driving around with human drivers for years before let loose and they are still highly restricted to city limits and local routes. And I was taking design courses in self driving control systems taught by General Motors back in uni in 2005.
I can’t reconcile the industry’s assertion that safety is the penultimate objective with our testing these things, uncontrolled, in densely populated urban areas.
More optimistically, at least it’s possible to diagnose these mistakes and work to mitigate them. The fact that we could now start treating auto collisions as seriously as plane incidents might be reason to hope.
The main criticism has always been that they were developing at a glacial pace.
What alternative path to deployment would you propose?
My concerns with Tesla are primarily about the safety implications of how it's automation capabilities get marketed, rather than the hardware itself… I also don't "get" the obsession over the "vision only" approach, which seems like an unwarranted compromise to me.
"Tech culture" intersecting with safety critical products and services indeed needs to be reckoned with.
The only argument is whether that's realistic.
Human drivers are still, mile-for-mile, the most dangerous threat to all pedestrians, blind and sighted.
The "run" part happened afterwards and should of course be punished harshly but the "hit" part seems to be her own fault.
But even if true, clearly the Cruise folks must have expected questions about the dragging and if none came should have wondered why. They witheld evidence, intentionally or not, but knowingly.
speechless here
* 10:30am: Virtual meeting with NHTSA. Wood shows full video, "again having internet connectivity issues causing video to freeze and/or black-out in key places including after initial impact" and again not bringing up or discussing pullover or dragging
* 11:05am: Cruise regulatory, legal, and systems integrity employees have pre-meeting for DMV and California Highway Patrol (CHP) briefing; Cruise team doesn't discuss pullover and dragging
* 11:30am: hybrid in-person and virtual meeting with DMV and CHP. Wood shows full video, again with internet connectivity issues and again not bringing up or discussing pullover or dragging
Internet connection issues, can happen to anyone
With stuff like this you want to be absolutely crystal clear what happened in your written and verbal reports. You can't just rely on a video that you may or may not have emailed over after previously showing it in such a manner as to obscure the key info.