What Self-Driving Cars Tell Us About AI Risks | Hacker News Reader

What Self-Driving Cars Tell Us About AI Risks

spectrum.ieee.org

18 pointsmfiguiere10 comments

The end-to-end systems Missy is describing here do exist, but none of the major AV developers are using them in current vehicles. What's concerning here is that Missy Cummings is someone who should be an expert here. She directs the Autonomy lab at Duke and served as a senior safety advisor to NHTSA on this topic, among others. She's been publishing on this topic for years, so it's hard to believe she hasn't read read any of the architecture papers, or looked at the numerous talks on the subject, or even just had a conversation with an engineer.

couldn't agree more. actually i don't understand how such a distinguished faculty can have such a primitive view of the industry.

While this does make some good points, I find it hard to take it too seriously when the author says things like:

> One failure mode not previously anticipated is phantom braking. For no obvious reason, a self-driving car will suddenly brake hard [...] The cause of such events is still a mystery.

There's no fundamental mystery about phantom braking. A self-driving car (or ADAS car with auto-braking) is 'simply':

1) Observing the world around it using whatever senses it's equipped with

2) Predicting the future state of that world and itself based on those observations, and

3) Controlling its output actuators as required to achieve desired outcomes and/or limit undesirable outcomes.

This is nothing controversial, this is the basic structure of any autonomous agent, from a toilet cistern to a kung fu master. So if a car 'phantom brakes' then it's perceived what it believes to be an imminent collision, and acted accordingly.

For simple radar-based systems, they can't really tell the difference between different vehicles or different lanes, so they guess based on the car's speed and any other speed readings they get with an intensity above some cut-off. That's why they don't work at low speed, because they can't tell the difference between other vehicles and stationary objects. These systems work okay-ish but there are tons of scenarios where they fail, and the system misinterprets its very vague sensory input as "car going much slower than us in front of us" and hits the brakes.

For more advanced systems that are doing vision etc. they're identifying objects and predicting how they'll move, and planning accordingly. This is usually better but still a Hard Problem (consider driving past a crowd of pedestrians, are any of them going to step out in front of you? Or consider a cardboard box in the road, is it just a box and safe to hit, or is it full of bricks?) so it's not surprising that current systems sometimes get it wrong. So do humans.

With all due respect, it seems to me like your simple explanation explains very little.

There is no way (besides kids having fun) that a human driver would intentionally hit a cardboard box found in the middle of the road, full or empty as it might be, and as well an automated system will brake or avoid the cardboard box.

But the point is that even when there is no cardboard box anywhere (nor pedestrians, nor cars, nor anything) these "phantom breakings" happen, example:

https://theintercept.com/2023/01/10/tesla-crash-footage-auto...

It is obvious that something went wrong in the sensors, in the tramission/processing of the data or somewhere there is a bug in the algorithms, but (AFAIK) nothing specific (and reproducible) has been found, in this sense "The cause of such events is still a mistery." seems to me correct.

Another article conflating AGI risks with ML risks. AGI has catastrophic possibilities. ML risks can range from accidents to annoyances.

Other than that, good insights from someone that works with self driving cars

I don't think self-driving cars are a major risk for catastrophe.

But I also am not sure we can say, in general, that ML is safe to the point that we can rule out catastrophe. Probably not-- but there's a lot of things we can't exclude. What if it completely poisons our discourse and ability to govern? What if it breaks labor markets and leads to profound unrest? What if it just plain screws up resource allocation and leads to famine? etc.

Ordinary mechanization and industrialization was pretty bad in a whole lot of ways (and the verdict is still out whether we will thread the needle and be OK or whether it will lead to catastrophe). ML and automation could arguably be a fair bit worse.

For the most part, ML doesn't control high-risk things with a human in the loop. The worst cases for ML so far have been stock market crashes from algo trading.

Its entirely possible that an update to self-driving car's algorithms cause a day of chaos as the self-driving cars lose control and crash. Worst case scenario.

I agree that the secondary effects of the ML systems are going to be far worse than the primary. We can only see how it goes.

> For the most part, ML doesn't control high-risk things with a human in the loop.

That was never my argument. (I assume you mean "without")

You said:

> > ML risks can range from accidents to annoyances.

Ignoring second-order effects isn't reasonable. Global warming is an indirect, nth-order effect from an industrialized economy based on fossil fuels.

It tells me that the world is going to completely suck in 50 years. It already does. You have about a 5-10% chance of speaking to an actual human for issues today, but everyday they are replaced more and more with "AI" driven automation systems. Now, we're trying to bring the "success" of that model to cars and the roads.

For automated driving, I still ask why? What is even the point?

Well the article is a mixed bag, but it certainly highlights human error is still the thing causing issues - humans pushing to profit from ai, humans racing to show off ai, humans legislating ai without understanding it, etc.

I don't feel like ai is going to save us from ourselves, somehow.

Reply on news.ycombinator.com