Hacker News Reader

Comment by largbae | Hacker News Reader

It is never that simple. Say the backup existed, and was global. By the time you get everyone briefed on how fubar it is and get agreement to load the backups, there are hours of changes from the unaffected customers that will be wiped by the restore, or have to be reconciled by hand for months. Sure, you can concoct the perfect antidote with hindsight, but their retro and next steps are sound.

View on HN

Build it properly. By definiton of following best practice IT IS or always should be not far from being able to follow this. If its not someone is to blame

What it means is that they never tested their disaster recovery system, because this would have been found right away. Or, someone would have reported it and an upper level exec would have signed off on it being okay to take 14 days to restore a small subset of users.

Again, not that simple. The customer restore procedure was almost certainly tested(and in active use as customers blow up their own data often enough). It was _not_ tested on 800 customer stacks simultaneously, as that was considered a sitewide disaster by whoever dreamed up the failure modes to test for. Meanwhile the actual whole site disaster restore plan may or may not have been tested, but it was useless for this case since some customers were unaffected and would be damaged by the whole site plan.

Then we are in agreement. Even if it were the case they had to do 800 separate recoveries (which I vaguely remember reading they couldn't do it individually), it means they never tested a large scale recovery situation and had no idea that 800 recoveries would take 2 weeks. That's a significant issue that they should have tested.

That lesson really stuck out for me also. My definition of “restore” has been too simplistic.

Not really. They managed closer to "undo" the break, which still makes it look like their disaster recovery might also be un-tested.

I'm sure there are many companies who would lose 24hr of tracking to get back online in a few hr vs 14days. In the real world with deadlines and commitments this is madness.

Reply on news.ycombinator.com