Hacker News Reader

Comment by cesarb | Hacker News Reader

I wonder how easy it is to really test these failover systems. Even if you disconnect the external power (which is risky, since you lose redundancy), it will be a "clean" disconnection, with the power going neatly to zero. I've found a few post-mortems of high-voltage faults, and the waveforms go crazy during the fault, which could lead to failure modes in the transfer switch which wouldn't be found on a "clean" disconnection.

View on HN

When we had the building UPS replaced several years ago, the contractors accidentally left one of the neutral bars disconnected... everything was up and working fine, the switch was flipped to cut off mains and ensure the backup system could take over, and suddenly there were pops and smoke all over the place as hundreds of desktops, servers, network gear, etc got a dose of ~430v instead of 240. That was a very expensive test.

That applies to other kinds of reliability testing too, especially everything that goes through a network.

It's easy to handle when something is completely unavailable (i.e. instant error), but when something, be it a database or some endpoint, is available but horribly slow, that's a whole different thing.

Seems like we need systems designed with "suicide" mechanisms built in, so that if they detect that they have a poor quality of life (err, I mean, that they're providing a poor quality of service) they'll shut down completely.

The circuit-breaker pattern is commonly used to make the affected system appear off-line to it's upstream dependents - you don't need to actually shut the server down completely and will probably want to log into it to see what's happening. If the dependent system can't function without the affected system, it should also indicate an error and further upstream systems should detect it.

In your house, you have the circuit-breaker pattern implemented in hardware. And upstream of your house there are many more layers of circuit breakers that generally increase in size until they reach a point where there is redundancy. A circuit-breaker going off in your house protects the other circuits in your house, a circuit-breaker going off on your street protects the rest of your neighborhood.

Industrial circuit-breakers are commonly a combination of hardware and software. Personally, I've lost more equipment due to brown-outs than any other cause. If you have equipment you don't want to experience a brown-out, program the breaker to cut off power completely.

You'd think that somebody'd start selling some kind of generator that you could hook up the systems to that would generate various forms of unclean power on purpose, precisely so that you could test those rare failure modes.

Reply on news.ycombinator.com