Outage Stories: The copy and paste outage | Hacker News Reader

Outage Stories: The copy and paste outage

wdkwwdk.com

48 pointskevin_nisbet10 comments

The way I read this, the redundancy mechanism depends on a process actually being dead when monitoring thinks it’s dead. But that’s really not the way you’re supposed to do this, is it? Maybe it’s just... resting. At the very least it seems like the allegedly dead process should be shot in the head before activating the backup one (although that’s likely quite difficult to do reliably even if the mental model is simple).

You're correct, it was a bad design. The vendor that built this system had a number of issues in this realm that demonstrated a lack of experience in building for the reliability targets of telecom. This was just one of the easier stories to tell.

STONITH is a good design option, but I think this implementation would've still struggled. The way they were tracking processes was fragile, so the shoot the process lost track of which pid it should signal. And this was a sort of clustered system, so if you shoot the node you take out other services (think of murdering a kubernetes node to terminate one process, you're going to inflict some spillover.)

We had a similar problem. Medium sized real-time trading firm HQd in the midwest. Had a bunch of dell stuff at our federally required DR datacenter. Dell tech (or whoever, I forget exactly, big company though) enters secure DR datacenter to put in a new blade in our blade server enclosure. I guess plugs a test router into our network via cat-5, then plugs in the power. Test router has the same static IP as some piece of Critical Infrastructure. Suddenly all of our real time trades are being routed to the Dell tech's test router, just into a black hole. All of our network config also gets routed to his test router in our DR datacenter. Everything stops. Even my personal workstation in our office, 10 miles away, my internet traffic gets routed to this guy's test router. Even our VOIP phones get routed into this black hole. 100s of thousands of day trades getting routed into the black hole that is his test router. Tech guy has no idea any of this is happening.

Finally after a few minutes,our head of IT/networking/everything finds the tech's personal cell phone, and calls him from his own personal cell phone, because VOIP is dead in the water now. Tech unplugs network device from network, within 5 min everything self-heals. Untold amount of fines and fees for not being able to handle trades during the outage.

Interesting. Yea we had a few of these due to I think it was vlan trunking protocol but I don't really remember. We had a cycle over 3 or 4 years where:

1. Configuring VLANs is hard, let's turn on the Cisco feature to sync VLANs 2. Network switch from lab or another network would be moved to production, with a higher revision index. All switches sync from the highest revision. 3. Outage happens because VLANs in lab are not same as production. 4. Investigation says vlan trunking protocol is dangerous, and should be disabled. 5. A year later, forget about the outage and repeat.

I think eventually there was a feature that sort of solved this problem, but I don't remember after all this time.

Often "zombie" systems/processes/hardware are much harder to handle resiliently than actually exploded/dead. You need end-to-end testing of the system to detect that even though everything "seems" right, something is wrong (and potentially even to decide that "box X is testing correct but not working, kill it".

My solution to this has been: Always have a dummy plug. I built this "on the fly" once by reconfiguring a service to use a blank copy of it's database, with permissions set to read-only. All the read queries succeeded, with zero results. All the write queries failed fast. It solved a hard problem fast.

This is why production services should have health checks that actually verify whether the service is doing its job in the system, not whether it's merely running the process. A process can be completely broken while still technically running, and vice versa (at least with janky ps-parsing scripts).

> And the way this system detected a process failure was to search the process list for a matching process. If we run the command super-ha-proxy..., and then it's in the process list, the process is working. If it's not in the process list, it must have crashed, exited, etc. and we need to launch a new instance.

> The supervisor script just starts detecting the old process, and has no idea that the additional instance has been launched

The number of times I've seen outages and weird flaky problems from some shitty script trying (and failing) to parse ps output...

daemontools, runit, s6, systemd, etc all solve this problem properly.

unrelated to whatever content was on that page, but for some reason I just don't trust websites that fade the text in; I also dislike pages like blogger.com that have a "loading page" intro to every site.

Just feels dirty or something.

It is interesting how the article talks about having to provide excellent uptime and then goes on to describe how the system was designed to be brittle. This is less about a copy/paste error and more about system design.

Reply on news.ycombinator.com