The Trollable Self-Driving Car | Hacker News Reader

The Trollable Self-Driving Car

slate.com

37 pointstptacek83 comments

I feel like the author of this article is placing altogether too much trust in human intuition and being very selective with their examples of which human qualities are beneficial to safety of the roads.

A self driving car will never distract itself. It can't get drunk. It won't be able to lose it's consciousness or mental acuity as it ages. It has access to huge amounts of real time information from sensors more accurate than any humans intuition ever could be and can act on that data in an instant.

If the only hurdles we have to this becoming a reality are edge cases like the ones listed in this article, then I am very optimistic for the future of self driving cars.

I think the whole point of the article is that the strengths and weaknesses of human and AI drivers are different, and that the differences are exploitable by adversaries, and may in fact be so easily exploitable that "adversaries" could include bored teenagers.

The sensor logs from the trolled AI system can show likely troll/malicious intent from the human. In case there's a dispute over who's at fault (& responsible for costs).

I will pack such a sensor log in my emergency aid kid and use it to reanimate dead people.

I'm sorry I don't understand this. I was talking in context of a human driver trying to force an accident against a machine-driven car. Is passenger death less of a risk when both drivers are human?

Bored teenagers can (and sometimes do) "exploit" real drivers. I've seen kids throwing rocks, and bricks off of overpasses, walking on the sidewalk lunge as if they are about to jump into the road as a joke, etc... I've also seen people slam on their brakes and/or swerve into other lanes to avoid empty plastic shopping bags blowing around, etc...

The invisible rope prank comes to mind.[0] I think there are a whole class of pranks to which AI will not be susceptible to. I assume the LIDAR can or will be able to clearly detect no such obstruction in the roadway (perhaps even if it does exist). However there are certainly many other hacks that will consistently fool AI, at least for a while.

[0]: https://youtu.be/G_pAcIjqcuY

He goes off on a tangent on the theory of the mind and saying that a computer might think a cloud is a truck.

In order to get an AI to think a cloud is a truck, you'd have to somehow reinforce the negative numerous times, essentially brain-wash the AI into thinking a cloud was a truck. Guess what else can be brain washed?

The author holds humans in far too high of regard here. Clearly given the number of accidents and road deaths a year, humans aren't very good at intuiting others' actions.

Bored teenagers are one annoyance. A greater annoyance is where automated car behavior is exploited to the detriment of inside occupant(s).

Think Brazil and how kidnapping and ransoming are not unheard of events. Unless these cars have an override, gangs and other criminals can exploit this "juvenile prank".

I remember a documentary that described a course for business executives in Brazil on evasive driving techniques for fleeing from kidnappers†. I imagine that all of these techniques are much riskier than any of a car's ordinary expected behavior. You could view this as another high-stakes variant of the trolley problem that several articles have identified as implicit in programming a car to respond to accident risks: is your self-driving car willing to drive in dangerous ways and violate traffic laws to escape from a dangerous situation? Is it willing to try to intentionally harm an assailant in self-defense? Is it willing to allow a passenger to override safety measures to do any of these things?

† To avoid giving the wrong impression about how Brazilians see crime risks in their country, I should say that I know many Brazilians and I don't know anyone who has ever taken this kind of course. I think it's easy for media to seize on and highlight exciting risks and exciting precautions: for example, I once also heard foreign media claim that "many" people in São Paulo commute by helicopter to avoid kidnappers on the ground. But friends in São Paulo cast severe doubt on this story, saying that the ultra-rich mostly like helicopters because they're fast and prestigious. :-)

He starts off by saying Humans are pretty good at guessing what others on the road will do. Driverless cars are not, then goes on to quote Our test driver, who had been watching the bus in the mirror, also expected the bus to slow or stop

So apparently in this case the human didn't out guess the car, since he saw the bus and could have stopped the car if he thought the bus would not stop. Humans are unpredictable, and no one (human or computer) can predict their actions every time.

The entire article was based on a premise that couldn't even be backed up with one good example. Granted, self-driving cars are in their infancy, but still.

Humans are horrible at guessing what others on the road will do. That guy up ahead, following 1.7 car lengths behind the other going 70mph... I have no idea what he will do.

Maybe a bee just stung his genitals, and he's going to careen off the overpass into traffic below. Or he's just another idiot, ready to dart into the other lane at 70mph, with even less distance behind and ahead of him than the 1.7 car lengths.

I can guess the category of actions he will certainly perform (general bad driving, stupid shit, etc.), but I have no psychic powers to know what he will do specifically.

And as for his reactions to driving conditions... for me to simulate those in my head, and come up with something plausible takes more time than it will take him to do it for real by which time it will be too late (and has probably sapped me of the brain power I needed to react myself).

It's a peculiar set of biases, confirmation and other, that let's morons say something like what you quoted.

The linked article [1] is actually far more worrying than Slate's concerns about bored 12-year-olds. If a $60 setup can do a rather precise spoof of lidar readings, we are open to scenarios such as:

- Causing the car to collide with other vehicles. E.g. getting into a fender bender to avoid the illusion of an impending frontal collision.

- Causing the car to stop where it shouldn't. E.g. imagine armed robbers or kidnappers waiting on a deserted stretch of the road with a lidar spoofing device.

- Causing the car to run people over. E.g. spoofing away a pedestrian on a crosswalk.

What's even worse is that all of these attacks would be difficult if not impossible to detect or track down. With zero risk you can acquire valid insurance claims, put victims in positions where they're easily robbed or kidnapped, or cause your enemies to be run over.

[1] http://spectrum.ieee.org/cars-that-think/transportation/self...

Couldn't you just shoot human drivers from a distance with a sniper rifle? I don't see why everyone is scared of new technologies being exploitable. Our civilization is built in the basis that most people are not assholes most of the time.

Sniper rifles are not easy to get and use.

I'd imagine if this became a critical piece of our infrastructure the FCC or some other government organization would step in and make products that interfered with them very difficult to obtain.

Not to mention I could have a rifle by the end of the day in my state.

Are they? A decent rifle is not difficult to obtain or use. There are many places you can practice. You don't need to be that accurate to be extremely dangerous and pull off an attack like that.

Meanwhile lidar spoofing attacks are likely going to be far harder. You need to obtain expensive (and probably illegal) equipment, and have the technical knowledge to use it. I don't know what the range of such an attack would be, but I don't think it would be better than a rifle.

Laser pointers are easy to get and point at helicopters and airplanes.

The FBI successfully arrests and prosecutes those who do. Why would this be any different?

https://www.fbi.gov/newyork/press-releases/2014/westchester-...

Getting shot with a sniper rifle is very clearly identifiable as a high-end murder.

Investigators arrive at car crash sites far more ready to consider them accidental.

Self driving cars would likely have black boxes that record everything that happened.

I don't know, a lot of these things seem to have analogs to human drivers.

I can cause emergency braking on the road outside my house by behaving like I'm about to run into the street while never leaving the sidewalk.

I could get on an overpass and shine a bright spotlight in drivers' eyes at night.

I get what the author is saying, and there will be a continual need to improve self driving cars. Sure there may be different ways to troll them, but I don't think he makes a strong case that they will be particularly more troll-able than humans.

I am more inclined to say that they are already good enough, and that this was probably the fault of the bus driver, as they tend to be VERY aggressive drivers that play a game of chicken with the cars around them every time they turn.

Really, the only question that differs usefully is whether self-driving cars are deterministic enough to be reliably trollable.

Which if they take off, we'll probably find "Yes, until the machine-learning algorithm at the top aggregates the metrics from multiple cars on the road, devises a new rule, and pushes that rule via auto-update."

It's more than just deterministic, but deterministic and stupid enough to make do some really undesired behavior. Using the same example as before, if I run a full speed toward the street, almost every car is going to screech to a halt. Doing this a rush hour, would be incredibly disruptive and probably cause a small traffic jam if no one made me stop.

The question with self driving cars will be, for example, can a group of people "herd" one into a garage to steal it, or perhaps get it to do something crazy like drive into a lake. My guess is at some point the car will just sit there, just like you would if a huge group of kids were trying to do this to you.

New rules like "ignore orange traffic cones," etc.

Imagine New York City -- where unless a car appears as if it is ready and willing to kill pedestrians people will jaywalk directly in front of it.

Now imagine New York City clogged up with self driving cars -- frozen at each intersection in fear of hitting someone. It'd be a disaster.

Humans will quickly understand that these cars are easily bullied, and many will take advantage of that. Why wait for your turn when you can just cut off a driverless car?

This topic comes up over and over again, and people keep missing a key difference between automated vehicles and human drivers---automated vehicles have cameras and perfect memories.

How many times do people get to play that game until the NYPD is flooded with metrics on where to go to hand out free jaywalking tickets?

They won't do that though, because they know that there will then be clamor for repeal of jaywalking laws.

I'm at 32nd and 2nd as I type this, and let me tell you: pedestrian pride is SERIOUS in NYC. We don't give a fuck about jaywalking laws, and if they start making this city impossible to walk in, city hall will hear about it.

Currently, jaywalking laws are (disgustingly) used to target people that police want to target and for no other reason. If you put it to a vote, I think that the vast majority of NYCers will prefer an "implied crosswalk" law, where pedestrians have the right-of-way in all situations except when they jump in front of moving traffic.

Let natural selection run its course. At some point humans will evolve an instinct to not walk in front of a moving vehicle.

Unless behavior changes, perhaps because something changed regarding enforcement of laws governing pedestrians which changed the perceived incentives. Which, were self-driving cars to become more common and, consequently, moving violations by human drivers less of a problem for traffic and safety and pedestrian violations relatively more of a problem for traffic, would probably happen.

> Humans will quickly understand that these cars are easily bullied, and many will take advantage of that. Why wait for your turn when you can just cut off a driverless car?

"bullied?"

This doesn't sound like bullying to me; it sounds like utilizing the roadway properly (whereas 'jaywalking' laws paint a picture of the roadway as belonging to vehicles, which we New Yorkers protest everyday).

If anything, driverless cars might make occupants more patient when dealing with pedestrians with whom they share the roadway. If you can chill and listen to music, read something, etc, maybe you won't be in such a hurry.

How long's it going to take before someone starts selling T-shirts with pictures of stop signs on them? What's going to happen when a self-driving car sees someone wearing one standing by the side of the road?

This is an awesome low-tech hack idea, but my sense is that if they can't distinguish a t-shirt from a road sign, there will have been other far more serious computerviz problems.

The state of the art is actually that the car can't tell diddly squat apart unless it's in the car's map database. So unless you hack the database and stand at the correct spot you can dress as any traffic sign you like...

This video from 2014 shows the software identifying a cyclist making a hand signal...

https://googleblog.blogspot.co.at/2014/04/the-latest-chapter...

Yes, obviously the car need to keep track of dynamic objects, but the algorithms are heavily dependent on a very detailed world map. It's a very long way until the three laws...

All the cops freed up from having to write speeding and parking tickets can hand out jaywalking tickets instead.

If cops start handing out tickets for jaywalking in NYC I can guarantee you there will be hell to pay. If there's one thing New Yorkers hold more dear than their bagels, it's their ability to jaywalk as they please.

Or the computer-controlled car could simply keep track of humans in its immediate vicinity and use any momentary gap that arises. It has a much much lower reaction delay than humans. Combine with electric engines and they can practically jump into any gap.

There will frequently be passengers in the car. Comfort puts a practical limit on how much stopping and starting you can do.

I'd like to work on Artificial Road Rage. There are some great opportunities, especially for networked vehicles. You cut off one Taxy™? Now there are 50,000 tons of mobile neural net wandering the city looking for ways to make your life miserable.

This has been a free market dream of mine. There's no need for a state to enforce minor traffic violations (and who knows what else) when "the network" is willing to do the enforcement. However this poses a huge problem when the rules are dictated by a central coercive entity rather than by consensus.

Don't forget people need ~250ms (1/4 second) to react to someone, self driving cars could easily operate at 1/10th that. So, a person needs 9 feet to stop at 10MPH, but a self driving car might only need 5 feet. If anything self driving cars will look like they are going to hit you far more than human drivers. http://www.random-science-tools.com/physics/stopping-distanc...

Also, people don't as a rule physically touch cars when jwalking. Thus, a self driving car rarely needs to stop even with significant number of people jwalking. At worst they slow down, but again less than you might think.

Simple: Just program the AI cars to randomly kill a pedestrian who jaywalks every million miles or so.

Oh wait. That's crazy-talk you say?

Well that's the exact same system we have in place right now

You don't even need humans for safety features to become a problem: Every day, I drive on a road that goes through the German woods. The road is barely wide enough for two average-sized cars to pass each other. If one car is wider than average, then one of the cars has to go off the road onto the dirt-and-stones shoulder.

The speed limit there is 60km/h. Nearly all the humans drive 80. If you drive 60 there, you will cause a queue to form behind you, because there is so much traffic on that 5km stretch of road.

The reason why I consider this to be such a good example is because there is no solution here for self-driving cars that will work in the real world.

If you just put self-driving cars as they are on that road, then you'll convert it into a giant traffic jam.

Of course, the only _logical_ way forward would be to widen that road so that it can actually sustain its traffic load, but that is not a real-world solution.

Because a) The local levels of government are too broke to fix all the roads like this even on a time scale of decades (!) and b) the road was built on the private property of local nobility in the 1920s (for weird historical reasons) and they have refused to give up the land for the road to be widened on at least one occasion in the past.

Now, this is Germany, possibly the most orderly car-nation on the planet. What about Italy? Or India? Or South Africa?

Self-driving vehicles will become ubiquitous on the Autobahns and in rectangular American cities. I would be very surprised if there were still many human truck drivers doing cross-Europe tours 30 years from now. But human-driven cars will not disappear in the lifetime of anybody reading this, simply because outside of a few well-planned environments, you have to bend the rules to maintain traffic flow.

Or, there might end up being fewer cars on the road because it's so easy to jump on a self-driving bus, then transfer at the station (with no waiting time) to a small pooled ride...

"Jaywalking" was originally a marketing campaign by the motor vehicle industry before it became law. Your description sounds like a return to the way it ought to be. The bully is the human driver who threatens to kill a pedestrian by being inattentive.

This is my biggest gripe when it comes to implementing self-driving vehicles. Once it gets to the 80%ish range and a lot of people can spot an self-driving car from a pedestrian-driving car, I just know there'll be so many people that will walk in-front of those cars because it's faster for the pedestrian than waiting to cross the road.

I do see a theoretical risk of induced accidents for the purpose of damage/injury claims. For example, jamming the LIDAR and using one's parking brake to stop abruptly without triggering the brake lights.

I think the trollable driver is significantly more difficult/dangerous. Humans have moods and anger issues and vengeful reactions to things.

As has been said repeatedly, accidents like running into a bus are innocent and could've happened to anyone. While the car was technically at fault, this likely wouldn't have happened had the bus driver been driving more defensively or even if the bus driver had been an AI too... lol

This is sort of the thing though - people can experiment and test the limits of a driverless system with little to no consequences. Once they know those limits, they can abuse them, knowing when it's always safe to cut off a driverless car for example. I can picturing all the videos on youtube now about how to fuck with these things.

With human drivers, that unpredictability factor adds to a level of fear which often prevents this sort of behavior.

I think there's also a primed number of people who would do this out of open hostility to self-driving cars, "took our jobs" types, etc and a lot of people who would do it just for fun or to make their own trips quicker.

> Still, people have an intuitive fluency with this kind of social negotiation.

Apparently, the author has never arrived at a four way stop at the same time as another car.

And yet - if there are human drivers, even if 4 cars hit the intersection at the same time, eventually someone will go first. It might be comical for a few moments, but eventually things work out.

With self-driving cars, you could end up in a deadlock and no one would get anywhere...

I imagine it should be trivial to work out an algorithm to break the deadlock. We have plenty of software systems, that needs to deal with similar problems.

From other articles I've seen this is something that's already been dealt with. One of the problems they had early on was all-way stops where human drivers were so aggressive that the self driving car would never get its turn to go, instead getting skipped by humans that just went when it hesitated for an instant.

As self-driving cars take off, they'll start to form a mesh network and communicate. This four way stop scenario is solved in an instant.

And when the grid is fully automated, they'll be a huge decrease in four way stops (and any slowdown/stop inefficiencies in general).

Driving 101. In the US, the car to the right has the right of way.

And I've been deferring to the left for 10+ years... Great.

Exactly.

The rules state that the person on the right has the right of way, yet people freeze up, then both go, then both hit the breaks, etc.

Be bold and cast fear from your heart mangeletti. Software cannot be bold. I intend to conquer self driving cars at all 4-way stop signs.

Law 101: driving laws are state-specific.

Law 101(a): Maybe I'm not understanding what you are trying to say here, but for CA and taken literally, it's incorrect. See, for example, CVC $21753.

http://leginfo.legislature.ca.gov/faces/codes_displayText.xh....

I think you've misunderstood me. I was responding specifically to the case pointed out by the OP in which 2 cars get to a stop sign at the same time. Here's the relevant CA code, and I haven't heard of a state that does it differently:

21800. (a) The driver of a vehicle approaching an intersection shall yield the right-of-way to any vehicle which has entered the intersection from a different highway. (b) (1) When two vehicles enter an intersection from different highways at the same time, the driver of the vehicle on the left shall yield the right-of-way to the vehicle on his or her immediate right, except that the driver of any vehicle on a terminating highway shall yield the right-of-way to any vehicle on the intersecting continuing highway.

http://law.onecle.com/california/vehicle/21800.html

We should pick an easier problem to solve for now. Let's have a few dedicated roads where we assist the technology either electronically or physically. We could automate major transport truck freight routes, for example.

http://ops.fhwa.dot.gov/freight/freight_analysis/freight_sto...

This system would pay for itself and provide immediate benefits. Convoys of trucks traveling at 75 mph, drafting to save fuel.

Arguably, that's already being worked on---semi-automatic cruise control that factors in signals about local traffic flow and the curve of the road has already been deployed in some luxury vehicles. Applying it to freight hauling probably has more to do with the entrenched cheap solutions we have now than with missing technology.

("Why don't we have both?" as the meme says)

So, in another decade or two we'll see convoys of 10 or 30 trucks cruising down the road sat 75mph, driving close enough to aerodynamically draft behind the other vehicles?

No, what you are saying isn't the same thing. People aren't going to be comfortable driving next to this. The risk on a separate "driverless" road is much less. And like I mentioned already, other redundant assistance can be added.

Building useful parallel infrastructure is expensive though and would probably be to slow. If current Autobahn construction projects here in Germany are any indication, building a parallel long-distance network would take longer and cost more money than waiting for automation to be reliable enough to share the roads. Even if we assume that there is space for that, which in many important places there is not.

Maybe some changes to make roads more robot-friendly (e.g. by adding robot-only exits, navigation infrastructure, ...).

I think the first step in that direction we are going to see on public roads are autopilots for trucks. Follow a human-driven truck, or maybe drive simple long-distance highway routes automatically. At the end, have a human take over again and do the city stuff. Either have the human on board but resting (which means they could at least take over with a few minutes delay if necessary), or have driver hop-on/off parking places at highway interchanges.

No one said that you have to build an entire parallel infrastructure. You simply find a section with a lot of traffic and duplicate that. It could be a long-haul section or it could connect a port to a train, for example. 68% of all freight is carried by trucks. The United States is big. Find somewhere where is makes financial sense.

http://selectusa.commerce.gov/industry-snapshots/logistics-a...

The automation technology can be further developed in a "real system". At some point it'll be ready for public roads.

Convoys of 10-30 trucks closely following each other at 75 mph on a dedicated right of way with few or no drivers?

Congratulations, you have invented a railroad.

1. Where do freight trains go 75 mph?

https://en.m.wikipedia.org/wiki/Rail_speed_limits_in_the_Uni...

2. Freight trains aren't going to eventually pull off the highway and drive themselves to the drop off in the towns and cities across the country.

3. Do you understand the huge cost difference between laying rail and creating a highway?

4. Why does everything need to be explained? Can't you put a little thought into it before posting. That urge to fire off snark is going to get you into trouble.

http://www.freightlinerinspiration.com/technology/

https://www.youtube.com/watch?v=CoPiQQLQB-A

How is this in any way better than a train?

No need for rails, regular cars can re-use the infrastructure. No need to move cargo to another vechile for the last mile.

Wouldn't the dedicated requirement mean that regular cars cannot use the infrastructure?

No need to move cargo to another vehicle can be a plus, but moving a container with a crane is quite easy and cheap.

This isn't a permanent solution, right? It's for several million miles of real world driving. You could certainly put other commercial vehicles on the same road. It's a commercial test bed for self-driving vehicles. FedEx, Amazon, UPS, etc might have other commercial needs. The traffic is separated for safety reasons at this time.

People forget that humans used to own the road. Sure there were horses and carts, but the horses were self driving and went around you if they weren't going too fast. Children used to play in the street, or at least the less busy back streets. It has really only been since the 1920s or so that humans got shoved over onto the sidewalks, or left out of the road planning entirely. A return to road traffic that deferred to pedestrians would simply put us back where everyone was in the zeroth through nineteenth centuries.

There are some places in the world that I'd hate to have a self-driving car, e.g. South Africa, Brazil, or West Oakland. In such areas, I'd assume it would be trivially easy to stop the car and rob the presumably well-to-do occupants.

How would a self-driving car react to situations like these? https://youtu.be/2VAlyDXl3LY https://youtu.be/9RzeRlhhprw

These are situations that you'd presumably want to run through the roadblock. Whereas in others, e.g. an undercover cop with a gun to your head, that you'd want to stop for.

> presumably well-to-do occupants.

Eventually, it should be cheaper to get a ride in one of these than a taxi.

Absolutely. I was just picturing something more like a Tesla Model S (which AFAIK is the closest thing to a self-driving car now available for purchase) but I'm aware of the push by Tesla, Uber, and others to make affordable AI transport practical.

" if you put a foot out into the road you might trigger a nemergency stop" .. you might trigger a similar painckey reaction in a human, that might include but wont be limited to , pressing brake, pressing throttle, steering suddenly... just let them develop it, and if you dont like it, dont use it. ^^ personally im glad people are working on this, especially for older people who get their licencces revoked due to bad eye sight etc., it can mean the world to them to be able to 'drive' somewhere.....

It's a possibly interesting question with a bad example case. In the specific crash cited, the human driver stated that they reached the same conclusion the car did. So to the extent that the car's algorithm was "gameable" here, a human was gamed the same way.

(Besides, since the "win" here was "Get the car to crash into your vehicle," I wouldn't call it "winning" precisely ;) ).

One thing I like about the self driving cards is that you can count on them to let you merge and not be aggressive jerks like some human drivers. I imagine this will cause lots of people to just cut them off though..

I really think that we often overestimate human intelligence. We may be able to infer what another person is going to do next but I don't think there is any magic going on.

PS: Try trolling humans, it works.

Reply on news.ycombinator.com