Comment by StreamBright | Hacker News Reader

Comment by StreamBright | Hacker News Reader

Parent Full thread

Yeah you are right. Also, there is another side of this coin. I used to work for a FAANG and here is an outage story for you. We had every angle covered of preventing user errors and stopping a single data entry to take down systems. We had reviews of changes, multiple approval required etc. One of my co-workers was working on a change that got approved he had to change the IP of a dns entry pointing to a load-balancer. The change involved inputting the IP address of the DNS server and the load-balancer's IP as well. Needless to say he mixed up the two IPs causing a several hour outage. The moral of the story is that you can architect the shit our of everything try to avoid these problems but there is always going to be a case that you did not cover and it can cause serious outages just like this one.

If manual entries are critical to a system, you can have one person enter the info and someone else to confirm the entry is correct. Not sure how common this is in other industries, but this was the method used at an extrusion manufacturing plant I once worked at.

Yea that’s what we call bureaucracy

I joke but the right amount of bureaucracy is good, but people are very careful about adding more of it because you can’t undo it

It is very common in aviation. That is why at any point in time one pilot is flying the plane and another one is monitoring what the former is doing. The pilot monitoring is responsible for verifying the actions made by pilot flying not to punish or blame anyone, but just to make sure that intended action (pronounced by pilot flying) is executed correctly. It is possible that, for instance, you wanted to set value to be X but accidentally rotated a switch a little more. There are also other responsibilities of course. But, yes, making action in pairs is something critical in aviation. That does not happen often when it comes to ground software though. You need to remember that NOTAMs are user-generated content, many systems which allows pushing data to these systems are older than me. Nevertheless database maintenance is possible to do in a safe way. It’s not a DNS nor BGP - it’s high level software where tampering files is absolutely avoidable.

That story sounds like poor architecture, is there no network mapping diff that would have shown the snake eating itself? Why not?

Reply on news.ycombinator.com