Hacker News Reader

Comment by aeturnum | Hacker News Reader

There was a great paper[1] I read about the human components in complex technical systems which argues that one of the roles of the human is to take the blame when the entire system fails. This does real valuable work for the companies involved and helps them avoid needing to answer the most uncomfortable questions.

[1] Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction by Madeline Clare Elish

View on HN

"Moral crumple zones" is an incredible description.

This goes for self driving cars just the same.

Yeah it leads to real problems for fully driverless cars, as there is no longer a human to blame.

Also note that blaming the human has been Tesla's strategy for avoiding responsibility on their software drive system failures.

Most car - pedestrian crashes in the USA never end up with the driver at fault. Not sure this will change when cars are autonomous.

Well I am talking about all crashes, not just car-pedestrian crashes. For example Teslas crash in to other cars and then the company blames the human driver.

An example used in the paper are the human drivers of Uber self-driving cars - one of which killed a pedestrian in Arizona.

The flip side to this is that every system involving software seems to inevitably devolve into a situation where the human is expected to no longer be responsible.

Oh, you floored it while the car was pointed at the wall? It has cameras, why didn't the car disable the go pedal?

This is happening more and more with cars, and it seems inevitable that it will happen in other spaces as well, as software is expected to protect us from ourselves.

There is a big difference between software being the safety net for humans, and humans being the safety net for software.

In the extreme, yes. But it looks like it is actually a continuum, with a shockingly blurry line forming the threshold.

Reply on news.ycombinator.com