Hacker News Reader

Comment by gautamcgoel | Hacker News Reader

It's amazing to me a single hardware failure can lead to this kind of chaos. Wasn't there any redundancy? Didn't anyone forsee this possibility?

View on HN

If you don't regularly test your failover, chances are it will not kick in when the primary fails. Especially if the primary is very reliable. Very common pattern.

Ideally, you periodically test your ability to failover. But if it doesn't work, well, there's a chance that you just caused a user-facing outage with your test.

Active->standby failover is usually that way because it doesn't support active->active. Which likely means that the active->standby system is bolted on with some 3rd party technology that isn't integrated with the application.

I've been involved with a number of failover systems where even when it worked there was the possibility that you might hit a _known_ condition that causes the fail-over to fail.

Pretty scary knowing the product your working on has a couple critical holes in the fail-over that management papered over, which while rare could happen. A lot of these solutions are the equivalent of pull the power on one machine move the disk to the other and power it on. The assumption being that the storage mirror/replication/etc being used to maintain transnational consistency for the "move the disk" part is actually going to be consistent when that happens.

This happened (or at least, was detected) an hour before trading opened. It should have failed over then. To me, that means that you could validly test your failover an hour before trading opened (or, perhaps wiser, an hour after trading ended). If it doesn't work, you learned without causing a user-facing outage.

The system is online, publishing market data and and accepting orders before trading starts. The exchange initially announced a delay in opening, and only later announced staying down for the day. It doesn't sound like they had the option to do what you described. The staff trying to resolve the problem was presumably doing everything they could before giving up for the day.

> If you don't regularly test your failover, chances are it will not kick in when the primary fails ..

What happens if the failover fails as well?

Failover should be tested, but what should be tested? There are many components and possibly innumerable failure reason is exists so 100% confident isn't possible.

Just like Chernobyl.

I have worked with Fujitsu in Japan before. I could see the following as one way it could happen.

There is a large disconnect with large system integrators there and the actual developers / architects; usually 2 or 3 levels of subcontracting. This means there are 2 to 3 levels of intermediaries taking a margin too, and as such, the actual developer doesn't have the fiscal wherewithal to push back on requirements or deadlines too strongly. (How do you save money on a 300k yen salary in Tokyo?)

If I was to guess, the requirement was there, but by the time someone technical got into the real nitty gritty, they discovered that the timeline was too tight to effectively do the testing. Instead of pushing back, they just rushed it through with a lot of overtime work.

Also, though maybe this is unfair, but Japanese culture doesn't work very well for software development. I worked with a couple on a project and they are absolutely unwilling to ask for help, ask questions, or proactively fix problems. Of course this is a very small sample, but I've heard very similar things from a friend who worked at a Japanese conglomerate for a couple years in Japan.

They treat mistake/bug/problem as a failure of a developer or engineer, not something to be proactively looking for and taking as a lesson for both engineer and code improvement.

From the article:

"When the error happened, the system should have carried out what’s called a failover -- an automatic switching to the No. 2 device. But for reasons the exchange’s executives couldn’t explain, that process also failed. That had a knock-on effect on servers called information distribution gateways that are meant to send market information to traders."

Automatic failover is like data backups, if not regularly tested it's like it doesn't exist.

Thats a bit extreme in my experience, I would say its just closer to luck

I would be very curious to hear about your favorable experience with untested failover systems. My last job we tested our active/standby failover like clockwork every 3 months. It served us very well.

My experience is that they worked when needed and it was a sigh of relief, and a little pat on the back that we perceived reality correctly when we had the foresight to set them up.

It's sort-of a saying in the industry that can be applied to many things. The biggest problem is the people who might know how to make it work are no longer there or have forgotten. If it's automated there's a chance it might work, but if it's slightly off then it's even harder to comprehend and adjust.

I've seen backups and failovers not work so many times that it's an amazing surprise when one actually works--usually after laborious manual off-script intervention and invention. I was being somewhat kind to failovers, I have seen smallish backups work on several occasions. An untested disaster recovery is the worst.

Edit: think of an unexercised procedure as not "It works for me" but rather "it worked for me once."

An untested failover system is indistinguishable from a box burning power and doing nothing.

On a failure, it might work. But there might not be a failure. Same difference.

Failover sounds like the sort of thing you just do all the time. i.e. at the extreme, you treat failover like loadbalancing.

These kinds of systems typically don’t use load balancing, they’re provisioned for the expected max capacity needed.

It seems like for true redundancy one should have multiple failover devices with regular actual switching between them.

I don't think failover won't work in every failure scenario, but won't work in this particular scenario. We can't test all possible failure situation regularly.

couldn't explain = didn't want to :)

But anyway, it also shows that the failover was not properly tested.

Or that the fault killing the first on also killed the failover? Like a rampage JIS encoding control character running over the wires.

It was a memory error. Hardware failure. The failover process is what went majorly wrong here.

It seems they had a press conference with more details in Japanese.

Someone speculated that since this is a RAM failure, the storage in question might kept crawling with reduced RAM and full workload so failover didn’t kick in(either failed from high load, or didn’t trigger from the fact that faulty RAM is successfully isolated). Sounds plausible.

RAM failures are fun. Maybe 15 years ago, I had one box reboot itself and come back up with only ~ 16 MB out of an expected 1 GB or so; it was pretty overprovisioned and was able to sort of keep up working from swap but was sending alarms on a system that was always quiet (near idle cpu normally, important but low throughput system). More recently, I've had a couple systems with so high a rate of correctable ECC errors that machine check processing ate nearly all the cpu; but it still had enough to pass healthchecks, but not handle requests in a timely fashion, but didn't manage to get an uncorrectable error that would have paniced the machine so failover could work.

Reply on news.ycombinator.com