Hacker News Reader

Comment by IgorPartola | Hacker News Reader

This is what's going to happen. If you've ever seen a machine learning algorithm in action, this isn't surprising at all. Basically, they'll behave as expected some well known percentage of the time. But when they don't, the result will not be just a slight deviation from the normal algorithm, but a very unexpected one.

So we will have overall a much smaller number of deaths caused by self driving cars, but ones that do happen will be completely unexpected and scary and shitty. You can't really get away from this without putting these cars on rails.

Moreover, the human brain won't like processing these freak accidents. People die in car crashes every damn day. But we have become really accustomed to rationalizing that: "they were struck by a drunk driver", "they were texting", "they didn't see the red light", etc. These are "normal" reasons for bad accidents and we can not only rationalize them, but also rationalize how it wouldn't happen to us: "I don't drive near colleges where young kids are likely to drive drunk", "I don't text (much) while I drive", "I pay attention".

But these algorithms will not fail like that. Each accident will be unique and weird and scary. I won't be surprised if someone at some point wears a stripy outfit, and the car thinks they are a part of the road, and tries to explicitly chase them down until they are under the wheels. Or if the car suddenly decides that the road continues at a 90 degree angle off a bridge. Or that the splashes from a puddle in front is actually an oncoming car and it must swerve into the school kids crossing the perpendicular road. It'll always be tragic, unpredictable and one-off.

View on HN

Very little of what goes into a current-generation self-driving car is based on machine learning [1]. The reason is exactly your point -- algorithmic approaches to self-driving are much safer and more predictable than a machine learning algorithms.

Instead, LIDAR should exactly identify potential obstacles to the self-driving car on the road. The extent to which machine learning is used is to classify whether each obstacle is a pedestrian, bicyclist, another car, or something else. By doing so, the self-driving car can improve its ability to plan, e.g., if it predicts that an obstacle is a pedestrian, it can plan for the event that the pedestrian is considering crossing the road, and can reduce speed accordingly.

However, the only purpose of this reliance on the machine learning classification should be to improve the comfort of the drive (e.g., avoid abrupt braking). I believe we can reasonably expect that within reason, the self-driving car nevertheless maintains an absolute safety guarantee (i.e., it doesn't run into an obstacle). I say "within reason", because of course if a person jumps in front of a fast moving car, there is no way the car can react. I think it is highly unlikely that this is what happened in the accident -- pedestrians typically exercise reasonable precautions when causing the road.

[1] https://www.cs.cmu.edu/~zkolter/pubs/levinson-iv2011.pdf

Actually, because there's a severe shortage of LIDAR sensors (much like video cards & crypto currencies, self driving efforts have outstripped supply by a long shot), machine learning is being used quite broadly in concert with cameras to provide the model of the road ahead of the vehicle.

That is what the comment is saying. Of course the vision stuff is done with machine learning - that is after all the state of the art. But that is a tiny part of the self-driving problem. So you can recognize pedestrians, other cars, lanes, signs, maybe even infer velocity and direction from samples over time. But then the high-level planning phase isn't typically a machine learning model, and so if you record all the state (Uber better do or that's a billion dollar lawsuit right there) you can go back and determine if the high-level logic was faulty, the environment was incomplete etc.

I was responding specifically to "Instead, LIDAR should exactly identify potential obstacles to the self-driving car on the road." - LIDAR isn't economically viable in many self driving car applications (for example: Tesla, TuSimple) right now.

Then your comment is off-topic, because the realm of discussion was explicitly "self-driving cars equipped with LIDAR". Uber's self-driving vehicles are all equipped with LIDAR, as are basically all other prototype fully-autonomous vehicles.

How is it off topic when we're discussing "current-generation self-driving" vehicles?

It's a point of clarification that the originally listed study doesn't take into account, but which could be important to the broader discussion. Especially considering that while this vehicle had LIDAR, the other autonomous vehicle fatality case did not.

> as are basically all other prototype fully-autonomous vehicles

As I pointed out with examples above, no, they are not.

The vehicle involved in the accident has an HDL64 on the roof.

Is it true?

You can get depth sensing (time of flight) 2D camera Orbecc Astra for $150 or 1D laser scanner RPLIDAR for $300. Of course they are probably not suited for automotive, but for me even extra $2000 for self-driving car sensors isn't that much.

But that's the issue: identifying a pedestrian vs a snowman or a mailbox or a cardboard cutout is important when deciding whether to swerve left or right. It's an asymptotic problem: you'll never get 100% identification, and based on that, even the rigid algorithms will make mistakes.

LIDAR is also not perfect when the road is covered in 5 inches of snow and you can't tell where the lanes are. Or at predicting a driver that's going to swerve into your lane because they spilled coffee on their lap or had a stroke.

With erratic input, you will get erratic output. Even the best ML vision algorithm will sometimes produce shit output, which will become input to the actual driving algorithm.

> Or at predicting a driver that's going to swerve into your lane because they spilled coffee on their lap or had a stroke.

Neither are humans, and a self-driving car can react much faster than any human ever could.

I can see when the car in front of me is acting erratic or notice when the driver next to me is talking on their phone and adjust my following distance automatically. I don't think self-driving cars are at that point yet. The rules for driving a car on a road is fairly straightforward - predicting what humans will do -- that's far from trivial and we've had many generations of genetic algorithms working on that problem.

Self-driving cars could compensate for that with reaction time. Think of it this way: you trying to predict what the other driver will do is partly compensating for your lack of reaction time. A self-driving car could, in worst-case scenario, treat the other car as randomly-moving car-shaped object, compute the envelope of its possible moves, and make sure to stay out of it.

Normal cars could do this too. Higher end luxury cars already started using the parking sensors to automatically apply the brakes way before you do if something is in front of the car and approaching fast. If this was really that easy, then we wouldn't have all these accidents reported about self driving cars: the first line of your event loop would just be `if (sensors.front.speed < -10m/s) {brakes.apply()}` and Teslas and Ubers wouldn't hit slow moving objects ever. I suspect that's not really how this works though.

Exactly, with LIDAR the logic isn't very tricky. if something in front, stop.

More than that - if something approaching from the side at intercept velocity, slow to avoid collision.

You're handwaving away the crux of the matter: while for a human, the condition seems straightforward (as we understand that "in front" means "in a set of locations in the near future, determined by a many-dimensional vector set", expressing this in code is nontrivial.

> I won't be surprised if someone at some point wears a stripy outfit, and the car thinks they are a part of the road, and tries to explicitly chase them down until they are under the wheels. Or if the car suddenly decides that the road continues at a 90 degree angle off a bridge. Or that the splashes from a puddle in front is actually an oncoming car and it must swerve into the school kids crossing the perpendicular road.

Are you working on the next season of Black Mirror?

In all seriousness, my fear (and maybe not fear, maybe it's happy expectation in light of the nightmare scenarios) is that if a couple of the "weird and terrifying" accidents happen, the gov't would shut down self-driving car usage immediately.

I am definitely not. Their version of the future is too damn bleak for me.

Your fear is very much grounded in reality. US lawmakers tend to be very reactionary, except in rare cases like gun laws. So it won't take much to have restrictions imposed like this. Granted, I believe some regulation is good; after all the reason today's cars are safer than those built 20 years ago isn't because the free market decided so, but because of regulation. But self driving cars are so new and our lawmakers are by and large so ignorant, that I wouldn't trust them to create good regulation from the get go.

> except in rare cases like gun laws

They're still very reactionary in that, which is precisely why it isn't very effective when a subset of them do react: there are plenty of smart things that could get proposed, but the overlap between people who know what they're talking about and people that want the laws is exceptionally small, so consequently dumb, ineffective stuff that has no chance of passing anyway gets proposed. What does get proposed is a knee-jerk reaction to what just happened, and rarely actually looks systemically at the current laws and gun violence as a whole. Example: the Las Vegas shooting prompted a lot of talk of bump stock bans. Bump stocks are so rarely used at all, nevermind in violence, and they will generally ruin guns that weren't originally made to be fully-automatic very quickly if they're actually used for sustained automatic fire. Silly point to focus on suddenly. After the Florida shooting last month so much focused on why rifles are easier to obtain than handguns. And it's because overwhemingly most gun violence is handguns. Easily concealable rifles are already heavily regulated at the federal level for that very reason.

> Example: the Las Vegas shooting prompted a lot of talk of bump stock bans. Bump stocks are so rarely used at all, nevermind in violence, and they will generally ruin guns that weren't originally made to be fully-automatic very quickly if they're actually used for sustained automatic fire.

<off-topic> This is non-sense. Typical semi-auto are wayyy over build. Unless mechanical wear or explicit tempering of the disconnector, there is no risk whatsoever to fire thousands of rounds with a bump stock. Actually, plastic/wood furniture are more likely to burn/melt before the mechanical parts will actually fail. As worst, you might bend a gas piston, but the rifle will otherwise be fine.

The underlying reasoning behind the push against the bump stock ban is that it was basically a semi-auto ban, as you can trivially with a bit of training bump fire any semi-auto without a bump stock from either the shoulder or the hip with a mere finger. </off-topi>

>> you might bend a gas piston

Tubes on low- to mid- range civilian DE guns can burn out very quickly, and are in fact designed to do so long before you get damage to the more expensive parts of the gun - I've seen it happen in most of the cases (which are admittedly quite few in number despite how often I'm there) where I've seen someone using a bump stock at a range. In the most recent case I think the guy was on his 3rd mag and it ruptured. It was a M&P 15 Sport II, if I recall. Not a cheap no-name brand, but about as low-cost as you can get and missing all the upgrades in the version they market to cops. High-end ARs would fair better, I'd expect, but high-end ARs are again so rarely used for actual violence because they're usually only purchased by people shooting for a serious hobby in stable life situations. And honestly I feel the same people buying those probably feel bump stocks are tacky and gaudy like I do.

Even in the most liberal interpretation of the proposed law, I don't think any bump stock ban would become a semi-auto ban. I could see the vague language getting applied to after-market triggers, especially ones like Franklin Armory, but you've gotta have some added device for any of the proposals I've seen to even remotely apply.

Largely because of Ralph Nader.

It's amazing how reformers get demonized even after their platform is accepted wholesale by the rest of the world.

US lawmakers are very reactionary in the case of gun laws too, it's just that gun owners usually have enough political pull to block them from successfully getting laws passed. (The current campaign for gun control is 100% a reactionary response to whatever's been making the biggest news headlines. For example, the vast majority of US gun homicides are carried out with handguns, yet gun control supporters seem to think it's absurd they're more tightly regulated than AR-15s - which are relatively rarely used to kill anyone and have more mundane uses for things like hunting - just because the AR-15s are in the headlines. The US's most deadly school shooting was done with handguns too.) In fact, I'd argue the reactionary nature of US lawmaking is important to understanding why "sensible", "common-sense" gun control laws are so strongly opposed in the first place.

> US lawmakers tend to be very reactionary, except in rare cases like gun laws.

And for a good reason, they are constitutionally prohibited to .

> the reason today's cars are safer than those built 20 years ago isn't because the free market decided so, but because of regulation.

All safety functionality was introduced and used way before regulators even knew that it's possible.

Edit: please explain the downvotes, ideally with examples

> Are you working on the next season of Black Mirror?

In Black Mirror, all cars in the world will simultaneously swerve into nearby pedestrians, buildings, or other cars.

It doesn't even need to be that. Imagine the shit show that a city would be when it's entire transportation fleet is immobilized because someone has messed with their safety features.

That could happen during a massive, remotely-triggered software update.

Just imagine a batch of cars with malfunctioning inertial sensors (it's brought down more than a few of my drones). GPS and perception (through ML or LIDAR) will work most of the time to override such errors, but if there was a second malfunction... "The car is swerving left at 1m/s; correct right."

Like the one that happened some days ago with Occulus helmets.

That could happen because of a malicious 'time-bomb' placed by a hostile state actor in such an update.

More likely it would be something about dissidents being killed in their cars, or cars going after people who aren't liked.

If that's bad, what happens when the robocars get hacked?

Like we have botnets made out of thousands (millions?) of compromised computers, we could have entire fleets of compromised cars, rentable on the black market using cryptocurrency, that could be used to commit crimes (e.g. homicide) while keeping the killer anonymous.

Scary stuff. I hope these self-driving cars will be able and designed to work while completely offline, with no built-in way to ever connect to a network. But given the biggest player in the field seems to be Google, they'll probably be always connected in order to send data to the mothership and receive ads to show you.

"I hope these self-driving cars will be able and designed to work while completely offline, with no built-in way to ever connect to a network."

Don't hold your breath about that. There will be a huge load of data ready to be sold to advertising companies just by listening what passengers talk about when passing near areas/stores/billboards/events etc.

> receive ads to show you

I'm just envisioning a scenario where the car automatically pulls to the side of the highway, locks the doors and dishes you with a 15 second ad, and then the doors unlock and the journey resumes as normal.

Or just (virtually) replace billboards with personalized content

Using these cars to commit homicide was actually one of the plot points in the second book of Three Body Problem trilogy by Liu Cixin. Very much recommended if you are into sci-fi.

Those books are interesting in that Liu seems to get away with seriously portraying narratives that would be out of bounds in the "approved" popular culture of the West. Murder by robocar is a minor example, but others include portrayal of the inherent weakness of societies in which men are effeminate and the superiority of leaving strategic decisions to military authorities. (I don't particularly agree with those propositions, but they are certainly present in the books.)

I'm half-way done with the last book in the trilogy and I am finding the assumptions and viewpoints of the world from a Chinese view quite interesting. The one that struck me most was how he presents humanities greatest strength over the technologically superior aliens is the human ability to conceal their true thoughts and the possibility of deception. Quite different from Christianity's high value placed on honesty.

The great stories of paganism and animism were composed by artists, and they all feature trickery and uncertainty. The Christian Bible has some of that (I like Job), but the majority was written by humorless unimaginative prudes. I suppose some of the Chinese philosophers are a little better than St. Paul, but mostly when they're being playful.

Confused by the downvote here. This is a perfectly legitimate question, and indeed it should be asked more, not less, often.

This is stupid question. The answer is - the same as if someone will physically mess with your car.

Physical hacking can only happen to one car at a time.

What if someone hacks the assembly line for Ford?

What if someone put a bomb in your car or mess with your breaks?

Right, but what if someone could put a bomb in 100k cars or mess with 100k car's breaks remotely over the internet. A large enough quantitative change becomes a qualitative change.

To be fair, all the examples you gave could also happen to a human driver.

will never happen. least not with any explicit depth sensors. any car with lidar has depth perception orders of magnitude better than yours and would never chase an object merely because it resembles road markings

I mean, maybe the self-driving car shouldn't exist if it's just going to run people over.

>So we will have overall a much smaller number of deaths caused by self driving cars

Why? This is what the self-driving cars industry insists on, but has nowhere near been proven (only BS stats, under ideal conditions, no rain, no snow, selected roads, etc -- and those as reported by the companies itself).

I can very well imagine a greater than average human driving AI. But I can also imagine being able to write it anytime soon not being a law of nature.

It might take decades or centuries to get out of some local maxima.

General AI research had also promised the moon once again in the 60s and 70s, and it all died with little to show of in the 80s. It was always "a few years down the line".

I'm not so certain that we're gonna get this good car AI anytime soon.

If self-driving cars 1. don't read texts whilst driving, 2. don't drink alcohol, 3. stick to the speed limit, 4. keep a 3-4s distance to the car in front, 5. don't drive whilst tired 6. don't jump stop signs / red lights it will solve a majority of crashes and deaths. [0]

The solutions to not killing people whilst driving aren't rocket science but too many humans seem to be incapable of respecting the rules.

[0]: http://www.slate.com/articles/technology/future_tense/2017/1...

But it doesn't work like that. You can't just say "If they don't do X Y or Z", because while they may not do X Y or Z, that doesn't mean they won't do A B or C that are equally bad or worse. Human and self-driving are two completely separate categories, you can't just assume that things one does well the other also does well, and so just subtract the negatives. You could easily flip your comment to go the other way: "If human drivers don't mistake trucks for clouds or take sharp 90o turns for no reasons then they're safer".

I do think that self-driving cars will be safer, but it's upon it's proponents to prove that.

As the sibling comment says, it does depend on self-driving cars matching human level performance. But with all AI/Neural Networks it is very possible to match human performance because most of the time you can throw more human-level performance data at it.

Each of the crashes that self-driving cars can be fixed and prevented from happening again. The list I gave are human flaws that will almost certainly never be fixed.

I further agree with you it's up to the proponents to prove that. It's a good thing to force a really high bar for self-driving cars. Then assuming the technology is maintained once AI passes the bar it should only ever get better.

> Each of the crashes that self-driving cars can be fixed and prevented from happening again. The list I gave are human flaws that will almost certainly never be fixed.

Not if you put neural networks / deep learning in the equation. This stuff is black boxes connected to black boxes, that work fine until they don't, and then nobody knows why they failed - because all you have is bunch of numbers with zero semantic information attached to them.

Neural Networks are only a small part of self driving car algorithms. The planning and sensor fusion etc. is usually not done with deep learning (for this reason). Only visual detection, because we have nothing else working better in this realm. But lidar, radar, sonar, what have you all work without any deep learning. The decision making on a high level is also without deep learning.

The only questionable parts will be where the vision system fails, and those are similar actually to human problems. Because human vision also often fails (sunlight on windshield, lack of attention, darkness, etc.)

> But with all AI/Neural Networks it is very possible to match human performance because most of the time you can throw more human-level performance data at it.

Are you in very vague words implying that AGI has been invented? AI might have matched humans in image recognition, but it is far away in general decision making.

And finally, I am tired of listening to "safer than a human". That should never be the comparison, but a human at the helm and an AI running in the background which will take over when the human does an obvious mistake -- you know, like a emergency braking system,

"Each of the crashes that self-driving cars can be fixed and prevented from happening again."

If those situations recur exactly as they happened the first time, sure they can be prevented from happening again.

That is, if a car approaches the exact same intersection as the exact same time of day, and a pedestrian that looks exactly like the pedestrian in this accident crosses the street in exactly the same way, with exactly the same other variables (like all the other pedestrians and cars around there that the sensors can see), the data could be enough the same that the algorithm will detect it at close enough to the original situation to avoid the accident this time.

But it's not at all clear how well their improvements will generalize to other situations which humans would consider to be "the same" (ie. when any pedestrian in any intersection crosses any street).

You missed a rather important point 0:

If self-driving cars are, at their best, roughly as capable as a human driver.

This is a big 'if'.

The solution to not killing people is a kind of rocket science. In fact, it's probably harder than rocket science[0]. It's predicated on a lot of things that are very, very, very hard. The fact is that humans, who are already pretty capable of most of these very very hard things, often choose to reduce their own capabilities.

If the best self-driving tech is no better than a drunk human, however, then we haven't gained much.

---

[0] though perhaps not harder than brain surgery.

I really don't think it's a big 'if'. As long as there is human level performance data, neural networks can be trained to match that level of performance. So it's a matter of time. It is indeed very, very hard, but also solvable.

I agree that it's solvable.

However, the process your describing, of collecting human-level performance data, requires the ability to gather all of the data relevant to the act of driving in a manner consumable by the algorithm in question. This is the simulation problem, and it's very, very, very hard (it's why genetic algorithms have traditionally not gotten much further than toy examples, in spite of being a cool idea). Perhaps it is the case that it is very important to have an accurate model of the intentions of other agents (e.g., pedestrians) in order to take preventative action rather than pure reaction. Perhaps it is very important to have a model of what time of day it is, or the neighborhood you're driving in. The likelihood that it is going to rain some time in the next hour. Whether the stock market closed up or down that day.

It also assumes that neural networks (or the more traditional systems used elsewhere) are sufficiently complex to model these behaviors accurately. Which we do not yet have an answer to yet.

So, when I say, 'a big if', I mean for the foreseeable future, barring some massive technological/biological breakthrough. That could be a very long time.

For those who do not get the reference: https://www.youtube.com/watch?v=THNPmhBl-8I

Well, one answer is either it will be positively demonstrated to be statistically safer, or the industry won't exist. So once you start talking about what the industry is going to look like, you can assume average safety higher than manual driving.

> This is what the self-driving cars industry insists on, but has nowhere near been proven

Because machines have orders of magnitude fewer failure modes than humans, but with greater efficiency. It's why so much human labour has been automated. There's little reason to think driving will be any different.

You can insist all you like that the existing evidence is under "ideal conditions", but a) that's how humans pass their driving tests too, and b) we've gone from self-driving vehicles being a gleam in someone's eye to actual self-driving vehicles on public roads in less than 10 years. Inclement weather won't take another 10 years.

It's like you're completely ignoring the clear evidence of rapid advancement just because you think it's a hard problem, while the experts actually building these systems expect fully automated transportation fleets within 15 years.

> It's why so much human labour has been automated. There's little reason to think driving will be any different.

Repetitive, blunt, manual labor now, and probably much basic legal/administrative/medical work in the near future. But we still pay migrant workers to harvest fruit, and I don't imagine a robot jockey winning a horse race anytime soon.

Driving a car under non-ideal conditions is incredibly complex, and relies upon human communication. For example: eye contact between a driver and pedestrian; one driver waving at another to go ahead; anticipating the behavior of an old lady in an Oldsmobile. Oh, the robots will be better drivers eventually, but it will be awhile. We humans currently manage about one death per hundred million miles; Uber made it all of two million. I expect we'll have level 5 self-driving cars about the same time we pass the Turing test.

> But we still pay migrant workers to harvest fruit

Harvesting fruit is far more complex than driving. It's a 3D search through a complex space.

> Driving a car under non-ideal conditions is incredibly complex, and relies upon human communication.

No it doesn't. The rules of the road detail precisely how cars interact with each other and with pedestrians.

> We humans currently manage about one death per hundred million miles; Uber made it all of two million.

Incorrect use of statistics.

> Harvesting fruit is far more complex than driving. It's a 3D search through a complex space.

Are you making a joke? "a 3D search [for a path that reaches the destination safely and legally] through complex space" is exactly how I would describe driving. (Also, driving is an online problem.)

Cars don't leave the road which is a 2D surface. In what way is that a 3D problem?

Ever heard of an elevated highway, ramp, flying junction, bridge, or tunnel?

I mean, yeah the topology is not as complex as a pure unrestricted 3d space but it's also more complex than pure 2d space. It's a search through a space, and it's complex, I don't know if nitpicking about the topology adds a lot here?

That's still 2D space. A car simply can't move along the z axis, so the fact that the road itself moves in 3 dimensions is irrelevant.

Even navigational paths that consider all of the junctions, ramps, etc. are simply reduced to a weighted graph with no notion of any dimensions beyond forward and backwards.

Your comment is just random hopeful assertions though...

>It's why so much human labour has been automated.

But how much human labour that is as complicated as driving has been automated? As far as I can tell automation is very, very bad when it needs to interact with humans who may behave unexpectedly.

>b) we've gone from self-driving vehicles being a gleam in someone's eye to actual self-driving vehicles on public roads in less than 10 years. Inclement weather won't take another 10 years.

>It's like you're completely ignoring the clear evidence of rapid advancement just because you think it's a hard problem, while the experts actually building these systems expect fully automated transportation fleets within 15 years.

Actually plenty of experts within the field disagree with you.

“I tell adult audiences not to expect it in their lifetimes. And I say the same thing to students,” he says. “Merely dealing with lighting conditions, weather conditions, and traffic conditions is immensely complicated. The software requirements are extremely daunting. Nobody even has the ability to verify and validate the software. I estimate that the challenge of fully automated cars is 10 orders of magnitude more complicated than [fully automated] commercial aviation.”

Steve Shladover, transportation researcher at the University of California, Berkeley

http://www.automobilemag.com/news/the-hurdles-facing-autonom...

With autonomous cars, you see these videos from Google and Uber showing a car driving around, but people have not taken it past 80 percent. It's one of those problems where it's easy to get to the first 80 percent, but it's incredibly difficult to solve the last 20 percent. If you have a good GPS, nicely marked roads like in California, and nice weather without snow or rain, it's actually not that hard. But guess what? To solve the real problem, for you or me to buy a car that can drive autonomously from point A to point B—it's not even close. There are fundamental problems that need to be solved.

Herman Herman, Director of the National Robotics Engineering Center @ CMU

https://motherboard.vice.com/en_us/article/d7y49y/robotics-l...

>>But how much human labour that is as complicated as driving has been automated?

Quite a lot actually.

These days you can produce food for several thousands of people using a few hundred people and plenty of machines.

Part of the reason why we haven't yet reached a Malthusian catastrophe is this.

Automated food production is very much simpler, because you're usually only producing one food item at large scale. That's the super easy stuff to automate.

Automated driving is more like a fully automated chef, that can create new dishes from what his clients tell him they like. Without the clients being able to properly express themselves. That's a lot more complicated than following a recipe.

Difficulty of automation goes roughly trains < planes << cars.

Automated trains are simple, but don't provide much value. Automating planes provided value because it's safer than just with human pilots. Automated cars are a different league of complexity.

> But how much human labour that is as complicated as driving has been automated?

Driving is not complicated at its core. Travel along vectors that intersect at well-defined angles. Stop to avoid obstacles whose vectors intersect with yours.

Sometimes those obstacles will intersect with your vector faster than you can stop, which is probably what happened to this woman. As long as the autonomous car was following the prescribed laws, then it's not at fault, and a human definitely would not have been able to stop either.

> Merely dealing with lighting conditions, weather conditions, and traffic conditions is immensely complicated. The software requirements are extremely daunting.

Which is why self-driving cars don't depend on visual light, and why prototypes are being tested in regions without inclement weather. Being on HN, I'm sure you're well familiar with the product development cycle: start with the easiest problem that does something useful, then generalize as needed.

> With autonomous cars, you see these videos from Google and Uber showing a car driving around, but people have not taken it past 80 percent. It's one of those problems where it's easy to get to the first 80 percent, but it's incredibly difficult to solve the last 20 percent. If you have a good GPS, nicely marked roads like in California, and nice weather without snow or rain, it's actually not that hard.

Right, so the experts agree with me that the problem the pilot projects are addressing is readily solvable, and that general deployment will take a number of years of further research, but isn't beyond our reach. This past year I've already read about sensors that can peer through ice and snow. 15 years is not at all out of the question.

Driving isn't just travel along a vector. Maybe trains, but not urban roads. urban roads are full of people and animals.

If a ball bounces in front of me, I slow down expecting a dog or a child running after it. No self driving car now, and in 30 years is going to be able to infer that.

Driving is essentially interacting with the environment, reading hand signals from people, understanding intent of pedestrians, bicycles and other drivers. No way any AI can do that now.

> Driving isn't just travel along a vector. Maybe trains, but not urban roads.

Trains travel along a straight line, not a vector in 2D space.

> If a ball bounces in front of me, I slow down expecting a dog or a child running after it. No self driving car now, and in 30 years is going to be able to infer that.

Incorrect. I don't know why you think humans are so special that they're the only system capable of inferring such correlations.

Or take a slow approach like Google Waymo which was almost perfect so far on the roads. Uber is rushing it, and this could cost whole industry.

exactly. I would personally never trust an Uber self-driving car, specifically because I've lost trust in the company itself.

Seems like you're jumping to conclusions here. Let's wait to see what exactly happened. I highly doubt that any of these companies just use "straight" ML. For complex applications, there's generally a combination of rules-based algorithm and statistical ML based ones applied to solve problems. So to simply: highly suspect predictions aren't just blinded followed.

Totally agree on your premise that we can rationalize humans killing humans - but we cannot do so with machines killing humans.

If self-driving cars really are safer in the long-run for drivers and pedestrians - maybe what people need is a better grasp on probability and statistics? And self-driving car companies need to show and publicize the data that backs this claim up to win the trust of the population.

It's a sense of control. I as a pedestrian (or driver) can understand and take precautions against human drivers. If I'm alert I can instantly see in my peripheral vision if a car is behaving oddly. That way I can seriously reduce the risk of an accident and reduce the consequences in the very most cases.

If the road was filled with self-driving cars there would be less accidents but I wouldn't understand them and with that comes distrust.

Freak accidents without explanations are not going to cut it.

Also, my gut feeling says this was a preventable accident that only happened because of many layers of poor judgement. I hope I'm wrong but that is seriously what I think of self-driving attempts in public so far. Irresponsible.

If you ask me which one is coming first, Quantum computing or "better grasp of probability and statistics" among the general public - I take the first with 99% confidence.

If a human kills a human, we have someone in the direct chain of command that we can punish. If an algorithm kills a person... who do we punish? How do we punish them in a severe enough way to encourage making things better?

Perhaps, similar to airline crashes, we should expect Uber to pay out to the family, plus a penalty fine. 1m per death? 2? What price to we put on a life?

>How do we punish them in a severe enough way to encourage making things better?

This is a tough problem, but if we stop or limit the allocation of resources to protect the secretive intellectual property that is autonomoilusly running people over, that is the most effective incentive I can see. Plus it's pretty easy to do.

We don't even have to force them to disclose anything, by affording less legal protection, their employees will open-source it for us.

It's not just about punishment though. It's about our brains being (mostly) good at saying "well they were unlucky, but I'd never find myself in that situation, so I'm OK." And with some brains wired for anxiety doing the opposite and instead only thinking about how they are the ones that would be in all those situations.

> maybe what people need is a better grasp on probability and statistics

Definitely, though my interpretation of your statement is "self driving cars have only killed a couple people ever but human cars have killed hundreds of thousands". If that's correct, that's not going to win anyone over nor is it necessarily correct.

While the state of AZ definitely has some responsibility for allowing testing of the cars on their roads, Uber needs (imo) to be able to prove the bug that caused the accident was so much of an edge case that they couldn't easily have been able to foresee it.

Are they even testing this shit on private tracks as much as possible before releasing anything on public roads? How much are they ensuring a human driver is paying attention?

Hm, people are fine with folks getting mauled in an industrial accident, or killed misusing power equipment. So its not purely a machine thing.

Maybe because its unexpected - the victim is not involved until they are dead?

The innocent bystander is indeed the reason. People working with machinery accept certain risks.

Pedestrians and cyclists ( and car occupants for that matter) accept risks near or on roads. You expect that at least 1. drivers are vigilant and cars are maintained so the 2. brakes don't fail or 3. wheels don't fall off. Yet in my life I have eyewitnessed all 3 of those assumptions being wrong at least once.

I'd be surprised if you could educate this problem away just by publishing statistics. Generally, people don't seem to integrate statistics well on an emotional level, but do make decisions based on emotional considerations.

I mean, people play the lottery. That's a guaranteed loss, statistically speaking. In fact, it's my understanding that, where I live, you're more likely to get hit by a (human-operated) car on your way to get your lottery ticket than you are to win any significant amount of money. But still people brave death for a barely-existent chance at winning money!

> These are "normal" reasons for bad accidents and we can not only rationalize them, but also rationalize how it wouldn't happen to us: "I don't drive near colleges where young kids are likely to drive drunk", "I don't text (much) while I drive", "I pay attention".

Tangent: is there a land vehicle designed for redundant control, the way planes are? I've always wondered how many accidents would have been prevented if there were classes of vehicles (e.g. large trucks) that required two drivers, where control could be transferred (either by push or pull) between the "pilot" and "copilot" of the vehicle. Like a driving-school car, but where both drivers are assumed equally fallible.

Pilots don't share control of an aircraft; the copilot may help with some tasks, but unless the captain relinquishes control of the yoke (etc.) he's flying it. So you'd still have issues where a cars "pilot" gets distracted, or makes a poor decision.

It's even more complex - there's the temporary designation "pilot flying" and "pilot not flying", with a handover protocol and whatnot: https://aviation.stackexchange.com/questions/5078/how-is-air...

> This is what's going to happen. If you've ever seen a machine learning algorithm in action, this isn't surprising at all. Basically, they'll behave as expected some well known percentage of the time. But when they don't, the result will not be just a slight deviation from the normal algorithm, but a very unexpected one.

Do we even know yet what's happened?

It seems rather in bad taste to take someones death, not know the circumstances then wax lyrical about how it matches what you'd expect.

May be for now, autonomous driving should be limited to freeways and roads that dont have pedestrian crossings/pavements.

This is a great point. Solve it one step at a time.

But the problem is Uber's business plan is to replace drivers with autonomous vehicles ferrying passengers. i.e. take the driver cost out of the equation. Same goes for Waymo and others trying to enter/play in this game. It's always about monetization which kills/slows innovation.

Just highway-mode is not going to make a lot of money except in the trucking business and I bet they will succeed soon enough and reduce transportation costs. But passenger vehicles, not so much. May help in reducing fatigue related accidents but not a money making business for a multi-billion dollar company.

That being said, really sad for the victim in this incident.

From what I could tell, it never was a money making business. Uber now and forever, operates at a loss.

Another quirk of people, particularly acting via "People in Positions of Authority" is that they will need to do something to prevent next time.

Why did this happen? What steps have we make sure it will never happen again? These are both methods of analysing & fixing problems and methods of preserving a decision making authority. Sometimes this degrades into a cynical "something must be done" for the sake of doing, but... it's not all (or even mostly) cynical. It just feels wrong going forward without correction, and we won't tolerate this from our decision makers. EVen if we will, they will assume (out of habit) that we won't

We can't know how this happened. There is nothing to do. ..and.. this will happen again, but at a rate lower than human driver's more less opaque accidents.... I'm not sure how that works as an alternative finding out what went wrong and doing something.

Your comment is easily translated into "you knew there was a glitch in the software, but you let this happen anyway." Something will need to be done.

Even if we assume that we wanted to address this for real, I fear that it will be next to impossible to actually assess whether whatever mistake caused this has actually been addressed when all the technology behind it is proprietary. I can easily see people being swayed by a well-written PR speech about how "human safety" is their "top priority" without anything substantial actually being done behind the scenes.

I think any attempts to address such issues have to come with far-ranging transparency regulations on companies, possibly including open-sourcing (most of) their code. I don't think regulatory agencies alone would have the right incentives to actually check up on this properly.

It‘s amazing how quickly things can happen after an accident.

In a nearby town, people have petitioned for a speed limit for a long time. Nothing happened until a 6 year old boy was killed. Within a few weeks a speed limit was in place.

Safety rules are written in blood. Often someone has to die before action is taken. But eventually people forget, get sloppy, or consider the rules completely useless. See: nightclub fires.

> So we will have overall a much smaller number of deaths caused by self driving cars, but ones that do happen will be completely unexpected and scary and shitty. You can't really get away from this without putting these cars on rails.

One of the big questions I have about autonomous driving is if it's really a better solution to the problems it's meant to solve than more public transportation.

Do you have any experience developing autonomous driving algorithms? Because you are making a lot of broad claims about their characteristics that only someone with a fairly deep level of expertise could speculate about.

Interesting insight. While self-driving cars should reduce the number of accidents, there are going to be some subset of people who are excellent drivers for which self-driving cars will increase their accident rates (for example, the kind of person who stays home when its icy, but decides that their new self-driving car can cope with the conditions).

Your comment reminded me of the One Pixel Attack [1] and my joke about wearing a giant yellow square costume...

[1] https://github.com/Hyperparticle/one-pixel-attack-keras

> Moreover, the human brain won't like processing these freak accidents.

I think this is really key. The ability to put the blame on something tangible, like the mistakes of another person, somehow allows for more closure than if it was a random technical failure.

Very well put.

It boggles my mind that a forum full of computer programmers can look at autonomous cars and think "this is a good idea".

They are either delusional and think their code is a gift to humanity or they haven't put much thought into it.

I don't believe I'm delusional, my code is certainly medium-to-okay. I've put a lot of thought into this. I think autonomous cars are a very good idea, I want to work on building them and I want to own one as soon as safely possible.

Autonomous cars, as they exist right now, are not up to the task at hand.

That's why they should still have safety drivers and other safeguards in place. I don't know enough to understand their reasoning, but I was very surprised when Waymo removed safety drivers in some cases. This accident is doubly surprising, since there WAS a safety driver in the car in this case. I'll be interested to see the analysis of what happened and what failures occurred to let this happen.

Saying that future accidents will be "unexpected" and therefore scary is FUD in its purest form, fear based on uncertainty and doubt. It will be very clear exactly what happened and what the failure case was. Even as the parent stated, "it saw a person with stripes and thought they were road" - that's incredibly stupid, but very simple and explainable. It will also be explainable (and expect-able) the other failures that had to occur for that failure to cause a death.

What set of systems (multiple cameras, LIDAR, RADAR, accelerometers, maps, GPS, etc.) had to fail in what combined way for such a failure? Which one of N different individual failures could have prevented the entire failure cascade? What change needs to take place to prevent future failures of this sort - even down to equally stupid reactions to failure as "ban striped clothing"? Obviously any changes should take place in the car itself, either via software or hardware modifications, or operational changes i.e. maximum speed, minimum tolerances / safe zones, even physical modifications to configuration of redundant systems. After that should any laws or norms be changed, should roads be designed with better marking or wider lanes? Should humans have to press a button to continue driving when stopped at a crosswalk, even if they don't have to otherwise operate the car?

Lots of people have put a lot of thought into these scenarios. There is even an entire discipline around these questions and answers, functional safety. There's no one answer, but autonomy engineers are not unthinking and delusional.

We look at the alternative which is our co-workers, and people giving us our specs, and our marketing teams and think 'putting these people in charge of a large metal box travelling at 100kmh interacting with people just like them - that is a good idea'...

It is not that we think that software is particularly good, it is that we have a VERY dim view of humanity's ability to do better.

    I won't be surprised if someone at some point wears a stripy outfit, and the car thinks they are a part of the road

Shouldn't be an issue with 3D cameras

You can't prove a negative and should be careful about promising what may turn out to be false. There is potentially quite a bit of money to be made by people with the auto version of slipping on a pickle jar. When there is money to be made, talented but otherwise misguided people apply their efforts.

Well, in that case a non-textured or reflective clothing could have a similar effect.

Wouldn't LIDAR pick it up still?

Imagine someone carrying a couple of two by fours they are holding vertically. They then stop on the sidewalk to check their phone at just the right angle. I am not really giving specific examples, as much as trying to illustrate that the way ML systems fail isn't by being slightly off from the intended programming, but by being really off.

Reply on news.ycombinator.com