Hacker News Reader

Comment by WJW | Hacker News Reader

That is an amazing story. Who knew that putting everything in a database without backups was risky?

Didn't they only say that their backups are fucked, but not that they didn't try to make them? I've definitely run into situations where the business is only willing to pay for the most rudimentary conception of a backup strategy, and then it turned out to have been subject to corruption.

Can't find it now, but I've often seen a phrase to the effect of "If you don't regularly test that you can restore from backups, then you don't actually have backups", and it seems to ring quite true here.

I'd go one step further - actually restore the databases and check that you can bring up the relevant applications that use those databases.

It's up there with "unless you test power supply failover, don't rely on that generator to save you".

I know of one case where power failed, the "24 hour" generators kicked in, power company says "We'll need 12 hours" and the outage happened 6 hours later when they ran out of oil.

I contracted at one place for a bit where they shut everything down in each of their data centres once a year and power everything back up.

When I was there this didn't go too well and they couldn't get one their data centres online again - failover to their other centres did work though. This was ~20 years ago in the finance sector.

That sounds like their testing worked out for them. Better than a random failure.

Yup. Have the problem when all the right people are awake and on-site to handle it.

I was in a building when someone inadvertently powered off the wrong equipment, which had been running for several years, and several of the power supplies failed to come back up. It was 1+1 redundant though, so we could quickly shuffle packs around to bring it back up without redundancy. Then, jogging through the building and asking if anyone had spares, we found a field tech in the lunchroom who had a pile of stuff in his van. Whole thing was back to 100% in less than an hour, and we let the beancounters sort out the field spares being used for office equipment.

If that same failure had happened during the overnight maintenance window (when volatile work was supposed to be performed), there certainly wouldn't have been the same resources around.

Backups were extant, but inaccessible, and non-functional due to causing a kernel panic when applied. [1]

[1] https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...

They said they knew they haven't been able to restore backups for 2 years. Which is another way of saying that they don't have backups at all.

Ah, didn't spot that

They explicitly mention that they have backups, but that the "backups are fucked", whatever that means.

Ransomware scenarios come to mind, where the attackers will look for and actively destroy the backups you have if they are remotely accessible with e.g. AD admin credentials, and only trigger the ransomware when the backups are gone. If you have offsite backups in some cloud system, that helps you against natural disasters but not against malicious activity. The need for backups that are not just offsite but also offline is somewhat recent and very, very many companies do not have them.

They backups are probably fine (as in “the data is in there”) but there may be no plans on how to restore them. Or they may be incomplete.

I doubt any external factors are to blame.

Reply on news.ycombinator.com