Hacker News Reader

That "engineering" you speak of? That's the source of the human error.

Any high availability system inherently aims for that, but "never" is a strong word. Sooner or later, you need to upgrade or replace physical hardware.

"The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future."

Sure now, but I wanted to know why they didn't do that from the beginning.

Why they didn't do what? "Increase the automation"? I suspect that they have been doing that from the beginning. It's an ongoing process.

I hate to quote Rumsfeld, but there are known unknowns, and unknown unknowns. Of course you want to eliminate the latter-- but there's (necessarily) no way you can ever know that you've done so.