Ironies of Automation (1983)

bainbrdg.demon.co.uk

54 points·lcuff··3 comments

> For very fast failures, within a few seconds (e.g. pressurized water nuclear reactor rather than an aircraft), when there is no warning from prior changes so that on-line working storage would also be useless, then reliable automatic response is necessary, whatever the investment needed, and if this is not possible then the process should not be built if the costs of failure are unacceptable.

This excerpt from the article's section 2.2 really struck me. I have seen a lot of software projects that were built, became mission critical, and only then experienced their first significant downtime. I have rarely seen organizations tackle the threat of failure first and design the system from that point of view.

This is a pretty good article. I like it, covers a lot of potential problems and offers good questions to ask about your automation activities and their impacts to overall capabilities.

> Perhaps the final irony is that it is the most successful automated systems, with rare need for manual intervention, which may need the greatest investment in human operator training.

This has played out, a number of aircraft mishaps are a consequence of this. [0] is a recent article discussing this.

[0] https://hbr.org/2017/09/the-tragic-crash-of-flight-af447-sho...

Yes, I've read quite a bit about AF 447. As someone who has taken some flying lessons, I know that in early aircraft, the controls were mechanically linked together. There is no way one pilot could push the pitch control forward and the other could pull it back. But they could on the Airbus with a 'glass cockpit' being flown. The result: Software averaged the result, and the elevator stayed neutral, with no indication to either pilot that this was occurring. Tragic poor design.

Reply on news.ycombinator.com