Hacker News Reader

Comment by averagewall | Hacker News Reader

That assumes the human won't mistakenly misuse the manual override and cause a crash when the computer would otherwise have been fine on its own.

Computers and humans are both fallible and it's not obvious that the human should be given higher authority than the computer. There isn't a simple answer to this kind of problem. Ultimately someone or something has to make decisions based on sensor data/vision/etc and that something or someone might always get it wrong.

View on HN

This reminds me of some dev tools (can't remember which ones) which require the user to do something like set an environment variable to "YES_THIS_IS_A_TERRIBLE_IDEA". If the switch to disable aircraft automation involves such cognitive barriers, it's possible that pilots could be relied upon not to use it except when truly necessary. Heck, the barrier could be that pilots aren't told of its existence until contacting maintenance. It sounds silly and useless but it would have offered greater assurance in this situation that landing could be done safely.

React is notable for this: inserting HTML strings directly into the DOM uses `dangerouslySetInnerHTML`, and also has used names like `__SECRET_DOM_DO_NOT_USE_OR_YOU_WILL_BE_FIRED`: https://news.ycombinator.com/item?id=11447020

I don't know how hard it was to do, but here's a case of some Apollo astronauts who injured and nearly killed themselves by unnecessarily using manual override and doing it wrong. I suspect they know full well what it means but still decide that a special situation requires it. Not so much like a careless oversight as in programming.

https://youtu.be/V-jRjyl7Fg4

The story starts at 47:50. I can't quite get the link to jump straight to it.

> Computers and humans are both fallible and it's not obvious that the human should be given higher authority than the computer.

I'm not saying the human should always be given higher authority than the computer. I'm saying that there should be some way, in a case where the human can tell that the computer is doing something obviously wrong, for the human to take higher authority in order to prevent the situation from worsening. Even if such an override were only possible in particular cases--for example, if a computer crashes or it is getting obviously wrong sensor data--that would be better than nothing at all.

Reply on news.ycombinator.com