Hacker News Reader

Comment by ChuckMcM | Hacker News Reader

Just one of a bazillion questions that comes up when your car is autonomous. What happens when the sekrit tech elite want to kill you and so they make your car drive off a cliff? Interesting questions to be sure and already being examined on such shows as "CSI" in the US.

Basically one of the reasons new technology rolls out slowly is that there are lots of people asking these sorts of questions and trying to have answers before they ship. Sometimes that works well, sometimes not (cite battery issues on 787).

View on HN

Another case worse than partially-hijacked trips was mentioned in a slashdot post, and if I recall correctly, there was a vague reference to the autonomous-car-makers considering it, and it has bothered me ever since.

Imagine a situation where a crash appears to the AI of car C1 to be be inevitable. But also, if C1 executes maneuver M1, its own occupants will be saved from harm, but another car C2 will have a worse crash or will crash when it otherwise would not - but if C1 does maneuver M2 instead, its own occupants will be killed or suffer worse harm, but the occupants of C2 will be better off. What if, in such a scenario, C1 would decide to "sacrifice" its own occupants to minimize the total harm (e.g., C2 has more occupants)?

Self-driving cars might well be designed this way. It might be mandated by law. Authorities would try to reassure objectors by saying the "avoidably killed by your car" outcome is very unlikely and that you're better off in the big picture of all risks.

Edit: On a closer reading I note that the linked page http://www.theatlantic.com/technology/archive/2013/10/the-et... covers, not exactly the above, but similar scenarios.

Apparently in the UK almost every inevitable crash will end with the driver swerving right, usually putting his passenger in harm (driver sits on the right). I'm told that's a natural and unavoidable reaction, so if it's true I assume the opposite holds for America and etc. I guess that's another decision the car has to make, do you hit an unavoidable obstacle head on or swerve, presumably drastically changing the survival chances of each passenger. The baseline human panic reaction is almost certainly easy to 'beat' but it's a horrifying though that a car would have to make that decision.

I've always thought that the protection part was a fallacy and the more obvious conclusion is that they swerve in the direction they are used to safely passing with.

It is likely that an AI sophisticated enough to make the consideration will be sophisticated enough to avoid the question.

That is, there isn't all that much reason for the cars to be operating on the precipice of disaster.

(If you want to say "But what about uncontrollable and unforeseen circumstances?", we should hope that we make the test out to be 'better than most humans', not 'perfect')

Assuming we change slowly to self-driving cars and they're operating for a while in an environment with human driven vehicles, no disaster scenario should be assumed out. They will have to deal with the situation that kill people right now, and them choices could easily reduce to "kill my passengers or those drunks in that car hurtling toward me" for example.

Better than humans is great, sophisticated AI is great, we can't expect perfect but I don't see a situation where they enter an environment much less dangerous than today's.

I expect the standard heuristic when the program detects an unavoidable collision will be to minimize impact energy, whatever that means.

(It won't have good information about the mass of oncoming traffic, but there aren't that many situations where choosing to collide with the oncoming vehicle makes any sense at all)

As with most thought experiments, it's a little far-fetched though. How often, in the run-up to a crash, is it really possible to make oneself safer (or less safe) at the expense (or benefit) of another car?

A more common scenario might be driverless cars that don't communicate with each other and therefore try to "optimize" a crash without knowing how other cars are going to respond – potentially far more dangerous than the magnanimous AI you talk about.

And imagine if the AI also rated the car occupants on their future value to society.

If we're talking about Google, I assume it would rank people based on the number of friends they have in Google+.

The cynical side of me suspects that the "Volvos" of the future will make this choice less often than the "Pontiacs". The rich will still have safer cars, AI style.

It seems like an interesting branch of "applied theology". Does one, in the end, have faith that these tiny entities imbued with lesser-god-like driving abilities really have one's best interests at heart?

You appear to be claiming this is a bad thing, when (assuming it worked as planned) this would make you safer, since on average you'll be in the other car.

Of course, an irrational fear of safer cars is already considered to be one of the big stumbling blocks for self-driving cars, even without this scenario.

http://en.wikipedia.org/wiki/Trolley_problem

I need to start carpooling more

Indeed. Here's a simpler one than the "sekrit tech elite" wanting to kill you...

What happens when the equivalent of the Mariposa botnet, but this time for autonomous cars, roots 12 million of autonomous cars?

But, this time, we have to believe that security is going to be done correctly, right? (this time, promised, no backdoors, no 0-days, no compromised servers, no compromised keys, etc.)

I does really hope this technology rolls out as slowly as possible because the security issues do really scare me.

oh god, ransomware for your car...

"We are going to kill 100 drivers every day until 1bn in bitcoin is sent to 1...x"

Just imagine the economic disruptions on day 3 when no one dares drive anymore

Didn't the first one already happen with Michael Hastings? Although to be fair, it's just suspicions and not any officially accepted story. Nonetheless, they're fairly strong suspicions, same as with the Karl Koch case.

Regardless of whether or not that is true (and full disclosure, I doubt that it is from what I've read) it is surprising to me how something with tenuous evidence can be presumptively believed to be true in certain circles. His own wife says it was a "tragic accident", and his brother says he believed Hastings to be in a "manic episode" at the time.

Why is it that random people on the internet have so much more suspicion and rage than his loved ones who were closest to him?

This isn't really meant to be directed at you, so much as expressing my frustration with a sort of casual canonization of a dubious event as a conspiracy.

Because the motive and the modus operandi make a good fit for the scenario.

The family is not interested in the emotional baggage of foul play, or the hassle of an arduous investigation, which would, hypothetically, probably be fruitless, even if suspicions were accurate. It's much, much easier for loved ones to swallow an official story of a tragic accident.

For total strangers, not burdened by emotions, the scenario is objectively disturbing.

I haven't been following the Hastings case, so I can only reply in general:

Following the recent Snowden disclosures, you should question yourself at disbelieving conspiracy so easily. You might be right (probably are, for all I know), but you have no reason to believe so other than pure faith.

Ask yourself this (and answer truthfully): A year ago - did you believe that the NSA is actively sabotaging standards, intercepting shipments in order to modify the software to be under their control, etc? If you didn't (and I'm sure you've had the conspiracies mentioned), then why on earth would you believe "the official word" on anything like remote control of a car?

You might have a lot of faith in the official status-quo for no reason at all.

I am fucking sick of people being surprised by the NSA revelations.

NSA malpractice has been documented all the way back to 1975 by the Church Committee's papers: http://www.aarclibrary.org/publib/contents/church/contents_c...

Then later on we had Clipper chips, ECHELON, Presidential Surveillance Program, TEMPEST, Total Information Awareness, DCSNet, Room 641A and plenty of other surveillance disclosures.

This whole stigmatization of conspiracy theorists as loons is wholly unfair, misguided and shows the speaker's ignorance of how intelligence agencies operate. Conspiracy theories aren't just idiotic pop culture ravings about Illuminati, reptilian men and ZOG.

Because conspiracies make for a good story to fill your hunger for curiosity, and when you are only loosely versed in the circumstances it is easy to find reasons to believe it was a conspiracy.

>What happens when the sekrit tech elite want to kill you and so they make your car drive off a cliff?

It's considerably easier to physically tamper with a car, such as the good old Hollywood standby of cutting brake lines.

Why worry about a fantasy of people trying to attack you remotely when there are plenty of real organizations that are much more likely to tamper with your car now?

> It's considerably easier to physically tamper with a car, such as the good old Hollywood standby of cutting brake lines.

It's only hard to exploit a serious security flaw the first time. Until they close it, it's relatively easy to exploit it thereafter.

The relative lethality seems likely to favour the software option too. Personally I doubt I'd get above 10mph before I tried using my brakes - (manoeuvring into and then out of my road.) Compared to, 'Wait until you're on the motorway and accelerate as hard as you can into oncoming traffic.' I don't see too many people walking away from the latter case - the closing speed would be something like 190mph, disregarding braking on the part of the oncoming traffic.

Because technophobia is harder when you don't have a technology to be afraid of.

Nearly exactly this happens in Stross's Halting State

You mean, bidding for a better bus route?

No, assassination by hijacking of automated cars

Reply on news.ycombinator.com