Cruise Knew Its Self-Driving Cars Had Problems Recognizing Children
theintercept.com
theintercept.com
Why bother with safety when a criminal disregard for safety makes you the world’s largest car company and the world’s richest person?
[1] https://dawnproject.com/dawn-project-safety-test-tesla-full-...
Edit: looks like Cruise is not long for this world. Insiders seem to have lost faith and are starting to leak. I'd bet it simply cannot withstand any serious scrutiny into its safety levels, practices, and culture.
> Koopman, who has a long career working on AV safety, faulted the data-driven culture of machine learning that leads tech companies to contemplate hazards only after they’ve encountered them, rather than before.
I haven't yet figured out how to effectively and efficiently communicate this mindset shift to folks educated primarily in ML culture. I am not sure I ever will. The closest I've come to an elevator pitch of the mindset shift is something like: "when human lives on the line and you're taking an absolutely massive number of samples IRL, doesn't it make sense to stop thinking in terms of analysis/probability and start thinking in terms of topology/nondeterminism? Ie, when you sample A LOT, unlikely shit happens. Manageable risks if you're selling ads I suppose. But not so acceptable if you're deciding whether a giant piece of unforgiving steel saw a bollard or your daughter."
[^1]: I read the quote-block in your post and immediately thought "Koopman". Then read the article and, sure enough, they're quoting Koopman. What a tragedy. The message was not only out there, but out there so loudly that the vague shape of the warning has a particular person's name attached to it in my mind. Yet, here we are.
No matter how many miles your car drives and how much data it collects, it will encounter a novel situation on the road. Unless it has higher levels of context / overriding safeguards / etc, a data driven only ML approach is going to fail dangerously.
One favorite example is the year old video of Tesla FSD attempting to unprotected left turn thru an oncoming trolley car while the center display 3D rendered the trolley car in motion. Clearly there is no overriding safety guardrail model above the path finding model. If the car can 3d render the object it is aware it exists.
And so we go on being perpetually "five years away" from self driving.
This is a common counter-argument, and sounds reasonable on face, and is even true in the limit. But we're not anywhere close to the limit. Reminding the dear reader of the actual context clears this point up quickly:
>> One favorite example is the year old video of Tesla FSD attempting to unprotected left turn thru an oncoming trolley car while the center display 3D rendered the trolley car in motion.
We're not asking for Nirvana. We're asking for not throwing up one's hands and declaring that perfect CV is impossible as one mows down pedestrians and trolley cars that one's CV system is clearly capable of identifying. For example.
BTW, just recently I needed to choose a CRC polynomial and discovered his work on that field as well.
(In particular: self-driving car companies are already highly motivated to not kill children.)
Transparency is sorely lacking in this sector.
Edit: removed a Cruise incident.
That was many years ago, but imagine if the accident had actually been reported and/or regulation existed then.
This should be expended to other industries as well.
In all engineering degrees it is impressed upon you that you are responsible for your work and if you behave negligently or outside your competency you can be held accountable for the consequences of your actions.
Did the car stop, notify the mothership, and then have a person direct it to pull out of the flow of traffic? How would we know? If the car moved on its own, that's bad. If the car moved after being contacting a "remote assistant agent", that's bad, too.
1 "The Cruise car ran over her, briefly stopped and then dragged her some 20 feet before pulling to the curb, causing severe injuries." https://www.nytimes.com/2023/11/03/technology/cruise-general...
I have doubts as to this being the right choice, but it seems less unreasonable than what I thought from the title.
The details are not that imporant. The main point is that they were aware they were couldn't match human performance on kid safety yet still put their cars on the street. all while talking about the terrible toll of car crashes on children.
> Two months ago, Kyle Vogt, the chief executive of Cruise, choked up as he recounted how a driver had killed a 4-year-old girl in a stroller at a San Francisco intersection. “It barely made the news,” he said, pausing to collect himself. “Sorry. I get emotional.”
https://www.nytimes.com/2023/11/03/technology/cruise-general...
Reality is nuanced.
I say this because in my personal tech circle, the guys I know most convinced autonomous cars are around the corner are literally the guys who do not drive. One is an unlicensed and within a decade of retirement, convinced he'll be able to use self driving cars by (early) retirement so he doesn't have to learn.
I literally could not have taken my first engineering job had I not had a car and been able to drive--and, indeed, would not have been able to go on many interviews. Had to go to remote job sites and, in any case, would have been that weird guy if I couldn't just rent a car and drive.
I could live in a city today and mostly remote work but I'd be very restricted in who I could reasonably visit and the activities I could partake in both locally and on vacation.
Knowing how to drive even in a city, because you might occasionally need to is a far more normie middle class thing.
I'm on a work trip in a HCOL city running $80/day in Ubers. If I was in a more cheapskate company they'd probably had made me rent a Hertz and drive myself for the week rather than blow $500 on Ubers. Knowing I'd never be put in that situation by an employer is another class signifier.
Some combination of prices and convenience have in my experience shifted the norms though. I'd have reflexively rented a car in past times for a routine suburban business trip by air. Today, I'll spend at least a few minutes considering whether I sort of need a car for convenience/personal activities or whether Ubers would just be prohibitively expensive.
I'd add that the inability to drive closes off a lot of options. I regularly do a lot of personal activities that wouldn't really be practical without being able to drive. It's not so much urban but rural activities.
Notice that this contains nothing of substance: no "human benchmark" is actually defined; nobody was asking if they constantly evaluate risks (do they also remember to breathe in and out every few seconds?); "the lowest risk tolerance" is meaningless, as is "the highest safety priority"; and obviously no vehicle will have "zero risk of collision", just as no vehicle has zero risk of being hit by a meteorite.
Even when reporting was mandated, they twisted the definitions to make them meaningless.
Cruise cars spend 2-4% of the time under remote assistance, but Cruise reported one disengagement per 95,000 miles in their most recent DMV report.
Why would you allow some tech company to develop its technology at the risk of pedestrian safety? What benefit does this have to the people living in these neighborhoods?
(Visalia is especially far, but I had a driver in SF tell me once that's where he drove in from.)
LA Times re: SF " On average, 30 people die and more than 500 are severely injured in the city's streets each year"
Cruise severely injured 1 pedestrian this year, so sounds like it's 2x worse than human drivers at least. Since there are 1000x more of them and they've only caused 500x the severe injuries. When Cruise causes its first fatality it will immediately be 33x worse.
A person playing Russian Roulette for 1 round successfully may extrapolate that there is zero risk. A person who has played 1000 rounds doesn't exist.
And humans get confused and have people pinned under their cars all the time. Nobody wants to continue running over a person (even if it’s the best course of action in hindsight), and they often get out of the vehicle to check before even noticing that someone is underneath.
Curious why they didn't share all the video if they think they did such an excellent job?
I know there's nothing that will ever convince you that autonomous vehicles are safe, but please think about what safe autonomous vehicles would look like if they were rolled out. These sorts of stories get tons of eyeballs, so journalists are trying to find any example of an autonomous system harming people. Even better if you can blame tech companies for being negligent or irresponsible. So every single incident, even if it's caused by a psychopath hit-and-run driver, is blamed on the machine. And every detail is exaggerated and misconstrued to make the machine (and the manufacturer) seem as terrible as possible. So the stories simply don't matter. The figure of merit is fatalities per passenger mile. Currently, Cruise's is zero.
Cruise employees are riding in these vehicles, as are their family members. They walk near these vehicles every day. If they actually believed they were unsafe, they'd deploy the vehicles far away from their homes.
Well, according to the article they are often human-driven.
Anyhow, Caltrans estimates 8.8 million miles traveled by vehicles in SF daily. That's 6.4 billion miles during the 2-year period that Cruise did less than 700,000 miles.
In California there's 1.38 deaths per 100,000,000 miles traveled. Cruise doing ~1% of the mileage in which one would expect a death to occur doesn't seem like enough of a sample to declare a comparative safety advantage.
If we cared about pedestrian safety, we wouldn't let ~half[1] the drivers on the roads drive.
[1] I kid, it's more like 5-10%.
We can address both things. The "but people are statistically worse" argument is a horrible, thoughtless "gotcha" in the form of whataboutism that makes light of people's lives.
It's just that nobody cares about enforcing low-level violations, and we just let them keep trying, keep rolling the dice, until their lack of consideration for other people kills someone.
It's also laughably unproven, unless it was to be scaled up, exposed to the full range of driving conditions, and measured for a long time.
But maybe it aligns with their vision of a free-for-all libertarian paradise, I am not sure.
Why people keep trying to use AI to drive cars, I have no idea. It’s like starting with the hardest of hard problems immediately. Fools errand, if you work in self driving cars get out ASAP while the rest of AI is still hot and you can get paid. Because the other stuff is on cusp of the trough of despair given the increasingly desperate emails from salesmen I get from XYZ.ai startups.
So AI entering a space changes a space, and humans respond faster to that change.
I think this is an underconsidered part of autonomous cars. As soon as every vehicle in Manhattan is autonomous, people will bully them into sitting idle at green lights while walkers, bikers, scooters and everyone else flagrantly jaywalk in front of them.
With a human on human interaction there is a chance the driver will try to kill you, or you feel guilt at inconveniencing them. With an AI trained to drive safely, the safest thing to do will accept being bullied.
A similar thing will happen in a mixed human & autonomous environment where human drivers will abuse the AI drivers.
Sounds like pretty much how things work now so long as anything has paused the road traffic--which is all the time.
Knowing there is no risk of a deranged or distracted driver hitting pedestrians will make them even more aggressive. People are underestimating how impatient a bunch of midtown commuters will be when they know theres nothing preventing them from continuing their multi block walk completely uninhibited by traffic signals.