Hacker News Reader

Comment by gambler | Hacker News Reader

>With self-driving cars, I think people kind of accept that now. That even if you don’t quite know how a self-driving car does it all, if it has a lot fewer accidents than a person-driven car then it’s a good thing.

Unless there is a condition in the environment that causes a systemic failure involving large number of self-driving cars at the same time. This happens all the time with other types of software. People shouldn't assume that machine-learned models are magically different and ignore all common-sense rules we apply to safety-critical applications. In your daily job, would you be willing to bet your entire car on a black box piece of code that seems reliable, because it worked most of the time in the past?

Also, recall that flash crash of the stock market. Imagine something like that, but with lots of cars on a highway. If you don't understand "how a self-driving car does it", you are unlikely to install appropriate safeguards.

PS: Please don't insult your intelligence by replying that we don't understand how people drive either, so it makes any opaque self-driving software okay.

View on HN

systemic failure involving large number of self-driving cars at the same time

This happens to human drivers every year. Snow, freezing rain, fog; these adverse weather conditions can lead to large numbers of collisions at the same time. Expecting self driving cars to never have a problem during rare environmental corner cases is to expect them to be vastly superior to human drivers all the time. If you're waiting for that day before you consent to self-driving cars sharing the road with you, you'll be waiting a long time.

>> Snow, freezing rain, fog;

Well, self-driving cars can currently handle none of the aforementioned conditions. That's why they are being tested where those conditions don't exist, and that's why we're about 20 years from widespread adoption. It rains and snows in much of the rest of the world.

It rains and snows in much of the rest of the world.

Yes, I live in the rest of the world and my family has been affected by fatal collisions brought on by adverse weather. I remain skeptical that self driving cars will be able to handle all of these things. I do think they'll be okay for mundane driving where the biggest risk is driver distractions, however.

I would like to see cars and maybe even roads made smarter in order to make driving safer. If all the cars on the road were part of a wireless network and they could signal to one another when they're braking or turning, then other vehicles could respond more quickly.

Heck, these systems could even be included in phones and pedestrian crossings so that cars could be signalled instantly when a pedestrian steps into the road in order to cross.

I used to think that this kind of system must be the future, but now I can only think of the greatly increased attack surface if my self-controlled car is not only taking directions from visual input and other sensors, but also all sorts of other unknown devices.

The system needs to be sane. The network should not be able to force your car to drive into a wall. Like any safety critical system, it should fail-safe. That means if the data it receives over the network would be unsafe to follow, given information from the other sensors, then it should discount that information as erroneous. This is something humans do all the time when, for example, reading a stranger's intentions.

But people are just too creative when it comes to exploiting systems. What if I force someone’s car to halt just under a bridge and then accidentally drop another car on top of it. Oops, a terrible ‘accident’! I realise this is a bit ridiculous but some other people are way more creative than I in this way.

What if I point a gun at someone and shoot them? As usual, the only thing that keeps you alive is that there are no motivated individuals that want to kill you. If there are, then they don't need you to drive a self-driving car.

You're ignoring the possibility of a mass-exploit, meaning the kind of cheap exploit that affects multiple vehicles at the same time, possibly killing a bunch of people, destroying infrastructure, or stopping the flow of traffic. Your typical adversarial "single pixel" change that makes your network believe unreasonable things, but in the real world. The existence of such exploits has already been demonstrated.

I think there is a potential for them to exceed human performance in all driving conditions. Machines can perceive and react much faster than humans, and once the perceptual and control problems are solved, all of them will become uniformly "good" at driving in difficult conditions.

BUT (and there's always a "but"), the problem is exponentially hard, and it can't be solved with today's technology.

I agree with you that widespread adoption will require modification of roadways, and potentially also segregation of autonomous and human-controlled traffic streams.

>these adverse weather conditions can lead to large numbers of collisions at the same time.

Cars sliding out of control on a snowy road is not a systemic failure.

If you have an entire fleet of self-driving cars using the same (or similar) technology, the exact same condition could cause all of them to misbehave in the exact the same way. This could happen simultaneously with a large number of cars. Or could be something that propagates in a cascading manner. (A car breaks, turns and spins out of control. The next car detects that, breaks, turns and spins out of control, etc.)

If you have no clear idea how your driving software works, how are you going to make sure this doesn't happen?

People have common sense, variability and adaptive learning to prevent things like that. Moreover, our traffic system, our cars and our driver education have been co-evolving prioritizing safety for the last 100 years. (Heck, some of it started with horses and carriages.) The complexity and size of the system have been gradually scaled up for just as long, and it was designed and refined to be used by humans.

It's astounding how many of AI enthusiasts fail to see these issues and differences. Traffic system isn't a board game.

> If you have no clear idea how your driving software works, how are you going to make sure this doesn't happen?

A lot of people don't know how typing "www.google.com" into their browser works but DNS "just works" for them. There is software in nearly every part of our lives that could systemically go wrong in catastrophic ways - we do our best to plan and prevent them.

> People have common sense, variability and adaptive learning to prevent things like that. Moreover, our traffic system, our cars and our driver education have been co-evolving for safety for the last 100 years. (Heck, some of it started with horses and carriages.) The complexity and size of the system have been gradually scaled up for just as long, and it's designed and refined to be used by humans.

Isn't that incredible? The complexity goes up: we adapt. As self-driving cars are introduced we adapt and evolve and incorporate these changes in technology, the driving "environment", security, safety measures, and laws and regulations.

> Traffic system isn't a board game.

I didn't see any efforts at self-driving vehicles just say "ahhhh! this driving stuff is so easy!" Of course it comes with huge difficulties, an enormous amount of complexity, a baffling amount of data, a ton of edge cases, and now a wealth of tribal knowledge on the subject.

When considering these things think of the systemic benefits as well. There was a story on the Google self-driving cars in Mountain View not taking a turn when the passengers of the car expected it would. Turns out that the IR sensors on the vehicle saw that a jogger behind a wall of hedges was about to enter the intersection to cross while jogging. Systemically implementing that change across a fleet of cars would then save untold numbers of injuries or lives.

We adapt.

How about: I like driving so don't take it away. or perhaps: Just because we can, doesn't mean we should.

> Turns out that the IR sensors on the vehicle saw that a jogger behind a wall of hedges was about to enter the intersection to cross while jogging.

Some cars today already have IR sensors (night vision systems) installed in them, and include pedestrian detection monitors as well. Cars don't have to be full on self-driving to have a large increase in safety and accident prevention.

>> A lot of people don't know how typing "www.google.com" into their browser works but DNS "just works" for them.

On the other hand, noone was ever run over by a mistyped URL.

To guard against the risk of a bug/design flaw affecting all vehicles simultaneously, I would include an independent "dumb" override that was too dumb to drive but could stop all vehicles as soon as any of several TBD conditions occurred, regardless of what the primary "smart" system was telling the vehicle to do. It wouldn't be "smart" AI, but just something listening to both the AI's smart sensors and it's own, independent, simple sensors, and if ANY of them reported a listed condition, it would override the smart system and stop traffic.

Human intervention would be required to enable the vehicles to move again.

After some real-world experience, I would want to give the emergency stop system the ability to release the hold on one layer of vehicles at a time, allowing the smart AI to drive again but at a speed limited by the "dumb" system to nothing faster than human walking speed, enabling them to clear the area. If they cleared without further incident, the next layer would be released until the whole stoppage either cleared itself or was restopped because something was still wrong.

Self driving cars do not represent a correlated risk. The situation you describe is completely fanciful. There is no plausible circumstance in which a large number of cars would simultaneously crash.

A 0-day vulnerability in their OS, exploited ...

Here's a link to a 100-car pileup: https://www.washingtonpost.com/news/dr-gridlock/wp/2018/02/0...

These kinds of crashes generally have the some root cause: everybody is traveling too fast for conditions, and when one thing goes wrong, the end result is that you get a chain reaction of everyone accreting into a single crash. If self-driving cars are capable of making the same misjudgements, then these kinds of crashes absolutely can happen.

Sure, but you can't criticize SDC's by saying that they have the same failure modes as humans. The comment I was referring to was suggesting that they'd have some sort of mass failure mode, in which some substantial percentage of all operating SDC's in an area would crash simultaneously. That is clearly not going to happen.

Yes outside of a few limited areas (mostly controlled access highways) and benign weather conditions we're going to be waiting a long time for level 4+ self-driving cars.

1) good self driving cars aren’t machine learning models. They leverage machine learning for certain tasks like object classification, but it’s not some black box neural net that learned how to get the car from A to B.

2) you’re describing how a system can crash. Driverless cars don’t share state, besides, I guess, vast scale astronomical and weather conditions. It’s not likely something would cause all cars to crash because of a collective hiccup. There may be a set of circumstances that consistently cause individual cars to crash, but that’s not really the same thing. And we can understand how these events are processed, so unlike people, we can adapt for future repeats of this scenario.

Driverless cars are part of a system that includes GPS, mapping, OTA firmware updates and updates about traffic conditions, regulations, weather, etc.

Errors or malicious attacks upstream could lead to disruptive or even dangerous behavior by a lot of cars at the same time.

Malicious actions are a concern, but they’re at about the same level of concern as alternative acts of terrorism. Unintentional errors in the things you list don’t feel like they’re huge threats. I would Imagine incidents with at most tens of cars and a relatively low risk at that. Not good, but not a scalable threat.

Some car companies are more conscious about this than others. Tesla has a refreshingly anti-fragile approach, where Waymo is heavily relying on data captured through massive amounts of infrastructure.

That sounds interesting. How is Tesla's approach anti-fragile?

This is a common conflation of intelligence evaluation criteria which is discussed in detail in Jose Hernandez-Orallo's 2017 book Measure of all minds.

It can be simultaneously true that we cannot describe in a causally deterministic way how a system works while also determining with some confidence that the system is performing as desired. This is true of all human systems.

So if you don't know how people drive, but have enough confidence to let them drive based on performance in evaluation scenarios, then similarly we should be able to let machines drive assuming they perform equally as well as humans over the same contexts without knowing how they drive either.

Not until they can demonstrate agency.

Do you want machines to have agency or do you want to control them? You can't have both.

Surely there is a continuum between total agency and total control.

Exactly.

Sure you can, just as we do with people.

You could divide this into two kinds of failures: one local to a physical area (such as a traffic light that somehow always looks green to the self-driving cameras) and global ones (such as virus, or a bug like a GPS time rollover causing all cars to do something crazy simultaneously.)

Local ones seem unlikely to cause casualties larger than events that fool large numbers of human drivers, like sinkholes or malfunctioning traffic lights or black ice on a freeway. Some of these will probably happen, and the systems will be improved, and they'll happen less and less often.

Global ones could be very large, and there are several instances of global, cascading failures in other systems like telephone or data networks. There are known ways to prevent many classes of failures, but getting to 100.000000% is hard.

> would you be willing to bet your entire car on a black box piece of code that seems reliable, because it worked most of the time in the past

For the average non-technical person, this is pretty much their relationship with all technology.

I don't think it's exclusive to tech either. I know lots of people that have absolutely no idea how the mechanical side of a car works, but they trust their lives to one at 100km/hr because it has worked generally pretty well in the past.

> I know lots of people that have absolutely no idea how the mechanical side of a car works, but they trust their lives to one at 100km/hr because it has worked generally pretty well in the past.

Car safety has made great progress in the past 10 or so years. People don't have to know how they work under the hood as long as designers and engineers already built safety into them and these designs have been heavily tested by third parties (e.g. Euro NCAP, IIHS, etc)

Self driving cars will have to prove their safety, especially in extreme weather and non paved roads.

Definitely! Everyone here agrees that they need to prove themselves effective, and there is more work to do there. I think Geoff's point that the parent poster disagrees with is that it isn't necessary to completely understand every single aspect of the model's weights in order to be able to empirically validate it as safe.

Every year something like 30,000 people are killed in car crashes in the US. Say the introduction of self-driving cars cuts that number in half and also say that of those 15,000 deaths, 2/3 of them are due to software or system errors, that's still progress, isn't it?

Also there's a lack of compelling evidence that self-driving cars have a lot fewer accidents than people do, and this argument is always worded in a deceptive way to kind of quickly shoehorn that assumption in and hope nobody notices. Even measuring this metric reliably is hard, and it's not helped at all by the fact that most of these cars in the real world rely on a human operator as a backup. How do we separate their responsibility? And where's the robust, independent review?

Any system that requires a human operator to passively observe a boring process for hours at a time, yet snap to attention immediately and without warning to take control and prevent human deaths is simply insane. That's the diametric opposite of how a safe system should be designed. If the human is required for safe operation, they should just be driving at all times to maintain vigilance. I don't see people talk about that much. The assumption seems to be that AI will quickly get better and the human back-up will not be needed--that the human is kind of a temporary workaround. Well I've seen plenty of engineering projects where the temporary fix becomes a long-term pain in the ass because the difficulty of the real solution was massively underestimated and under-budgeted. We might get stuck with these terribly designed AI-human hybrid systems for many decades as AI keeps getting pushed into the horizon and we slowly realize that this is one of those hard problems that doesn't progress the same as most other tech does.

There are pure driverless systems out there, but they don't cope well with many of the incongruities and fringe situations which are an unfortunate reality on the open road in most of our country. I think they will be confined to niche uses for a long, long time.

Despite your warning, I think it's important to say that we don't understand exactly how the brain works. Yet, we still allow people to drive. To know whether we should trust someone to drive, we created a test (driver's license test).

I don't see why this would be any different for deciding whether computers should be allowed to drive - we can just make sure self-driving cars pass a road test. We could even do better than driver's license tests and have a human driver at the wheel during the test just in case.

we don't just let people drive because they can pass a road test. There's probably 12 year old's that can pass a road test with enough training.

We let people drive because we know that they are fully capable agents able to make decisions even in case of failure. In other words, they are responsible, reliable, fully cognisant individuals, far beyond the technical ability of operating a machine.

A self-driving car is like a parrot trained to speak. Not only is it not an agent aware of its larger surrounding and potential implications of getting into unknown situations. It also has no real understanding of what driving is at all, it's even stupider than a pet. And because its software is usually deployed to thousands if not millions of devices at the same time, there is little to no variance to protect from extreme failure.

> We let people drive because we know that they are fully capable agents able to make decisions even in case of failure. In other words, they are responsible, reliable, fully cognisant individuals, far beyond the technical ability of operating a machine.

I wish I could share your optimism in humanity, but what you just claimed sounds far from being true universally.

Take your typical driver and throw in a failure situation they have never trained before (because, who has time to train them for every possible, or evens few likely failures), and it is not likely to end well.

>Take your typical driver and throw in a failure situation they have never trained before (because, who has time to train them for every possible, or evens few likely failures

Humans have a base line of common sense reasoning and the ability to create coherent models of the world. A human driver isn't (rare exceptions aside) going to drive up an impossible street or try to dodge a street sign mistaken for a kangoroo in West Germany.

There's nice examples of how an ML algorithm can be tricked into all sorts of pathological behaviour by choosing adversarial images, there's no baseline of reason to fall back on.

Imagine a model that has one of these fatal edge cases baked in and a thousand cars suddenly mistake a flash of light for a u-turn. Sure, one or two human drivers out of a ten thousand might drive that irrationally on occasion, but compared to a machine we can usually cope with this.

And I don't even want to imagine what happens if this turns malicious. Imagine a bad actor on a bridge above a highway with some artifact that messes automated cars recognition up and the developers forgot to put some hard fail-safe in for that piece of road.

Your point about agency is well made.

It's worth noting the downsides to this agency too, though: Agency is what makes the stereotypical male 20yo driver speed and drive dangerously to show off to his friends, which leads to a very large number of accidents.

It's unclear what accidents this agency avoids though. I can imagine a few situations where it helps (avoiding a terrorist attack by speeding away from the scene?). But in most cases the default operation for a self-driving car will be to stop as quickly and safely as possible in unusual circumstances, and it's unclear how often this is the wrong decision.

The vast majority of failures in automobile usage today are caused by human conditions that will never affect a machine. Alcohol, drugs, fatigue, social distractions, aggressive driving.

Google something like "top causes of automobile accidents". No list I found had exogenous causes (rain, snow, etc) in the top 5. For machine driving to top that isn't going to be that hard given that we are our own worst enemy on the road.

This is still a biased argument. Once it scales up, you'll be able to google "top causes of self-driving car accidents." They will have their own flaws and quirks that we don't anticipate yet, and we don't have any good reason to believe the death toll will be inherently less with computers than with people. Plus, there will be exogenous causes of accidents that are far down on the list for humans, but for AI it might represent a much higher risk and this changes the nature of that list entirely.

There just aren't reliable statistics on what the safety of this tech will be at this point in time. All we have is extrapolations with lots of bias and little independent review. Maybe it will eventually be worth it, but that experiment will come with a great many human deaths and we can't simply brush aside the ethics of that. I personally think it won't prove to be safer than humans for many decades. This is far more of a complex problem than a lot of the developers think, and I believe there's a case of engineering arrogance going on here as well.

On the surface it seems like we have something very close to a viable product, but the remaining little details and issues are important and represent an enormous amount of work, where we are going to see diminishing returns and enormous difficulty making a system that is flexible and robust enough to handle all the required situations more reliably than the average driver on its own for thousands of miles, scaled up to millions of units around the world. They will be good enough for niche applications, but the dream of a general self-driving infrastructure is pretty far off.

This is a straw man too. We don’t flip a switch and suddenly all cars are driverless, and then we discover that they can’t handle all situations and everyone dies. We slowly scale it out to environments where it is provably safe. We will not see “top causes of self driving car accidents” because known conditions will merit additional development or restriction on usage. What kind of thing could conceivably make that list? Something like snow doesn’t fit, because they won’t run in the snow unless they’re safe to run in the snow.

You don’t need to be better than humans for all situations. You need to be safe. Humans are generally well above that threshold, except for when they’re drunk, high, tired, distracted, or otherwise stupid. It doesn’t really make sense to compare deaths per mile for cars vs attentive humans in stable conditions. Both can honestly be reduced to the point at which they’re practically 0 unless it’s the victim’s fault. We do need to be attentive to edge cases that cause cars not to be safe, but we can also observe the overall pattern and draw conclusions on how likely unknown edge cases are to be encountered.

> Unless there is a condition in the environment that causes a systemic failure involving large number of self-driving cars at the same time.

This happens all the time already: there is a condition in the environment (eg when it snows in Seattle) there is a systematic failure involving large numbers of people driven cars at the same time.

People just don’t have experience (erm, “training”) with the weird unusual environmental condition and fail to deal with it. Not surprising that computers would have similar problems.

> PS: Please don't insult your intelligence by replying that we don't understand how people drive either, so it makes any opaque self-driving software okay.

But we actually do know how well people drive. We have data on failure rates for sure. We don’t know how they drive, that is completely opaque.

> PS: Please don't insult your intelligence by replying that we don't understand how people drive either, so it makes any opaque self-driving software okay.

I won't insult my intelligence; I'll ask why you trust people even though you don't know how they work and you know that at a high rate they intentionally impair themselves before driving?

Recall that time when a human trader lost a billion dollars?

https://traderhq.com/traders-cost-company-billions/

Remember when the economy tanked because everyone in finance used Black-Scholes model that seemed reliable, because it worked most of the time in the past?

why do people (not you, the quoted part) keep talking as if self-driving cars have already reached the point where they are safer than humans?

Good point. The answer is probably that nobody really cares about safety, they just want to have their cool toys right now. If people really cared about safety, they 'd be advocating for the only certain way to reduce car accidents: fewer cars (or in other words, more public transport, more trains, etc). Except that wouldn't make sense because the whole point is to sell more cars. Self driving "safety" is just the latest marketing gimmick.

So, this reminds me of a nuclear meltdown vs. the constant pollution form a coal power plant nearby a city. Even with one of those single, ugly, massive failures they could still very well cause fewer deaths.

I think your concern here is less about flaws in the AI training and more about scale issues when you have a million cars with the same brain and thus the same flaw. I share your concern. Human brains have lots of flaws too, but at least every human driver has a different brain, so some fraction of mistakes cancel out.

But another way to look at this is the fact that "cloning" the same AI brain across multiple cars is equivalent to a single AI brain controlling many cars. And that does have human analogies where you have a single person who has a dramatic amount of power. Any flaws in that human's reasoning can be disastrous.

Trump having access to the nuclear launch codes comes to mind.

In practice, the way we mitigate this is by surrounding that human with advisors and other brains to try to help shore up deficiencies. We could do something similar with AI. Have a couple of differently-trained models and compare their outputs. If they disagree wildly, then try to gracefully shut down the system.

Reply on news.ycombinator.com