Hacker News Reader

Comment by w8whut | Hacker News Reader

daily test restore is infeasible for anything but toy projects. You should periodically test your restore procedures, but its incredibly costly and time consuming for sizeable platforms. Its just not that easy to restore a 10+TB backup for example, and thats a _tiny_ backup size for a b2c product.

they can easily go into the hundreds of TB, depending on your platform.

and i might add: i vividly remember gitlabs article how they have had automated backups and test restores for years, but when they actually needed them... it turned out some data wasn't part of it after all. just because youre testing your restore procedure doesnt mean you've actually accomplished anything.

View on HN

Have two backups, the most recent data and everything else. Archive data to different databases, e.g. only have the most recent 6 months in a production OLTP database.

"daily test restore is infeasible for anything but toy projects. "

It probably depends on what you call "toy" project. If you work for Google, yes I think everything is a toy project, and you're right. I only worked for ~$200M ARR/1M DAU businesses and restoring was no problem. From your point working for a FAANG business it's a toy project I can see that. But there are many more "toy projects" of this kind than FAANG companies.

"10+TB backup for example, and thats a _tiny_ backup size for a b2c product."

Sure.

> but its incredibly costly and time consuming for sizeable platforms.

If your restores are too time consuming to test regularly, they sure as shit aren't going to be useful in a disaster.

Why is that? Hours or even days of downtime are still better than just losing all data. It's a simple cost-benefit analysis, and it's ok to pick different trade-offs depending on your use case

I have spent most of my career in newspaper publishing and banking.

A newspaper that doesn't publish for a few days might recover. A bank that drops off the Swift network for days isn't a bank any more.

Russia and Iran would like to have a few words with you.

Banks regularly close for multiple days for bank holidays. Unscheduled downtime is a somewhat different story, though.

Luckily, the traditional SWIFT/banking infrastructure is so negligible these days, my phone can host a classic banking infrastructure for an entire small country.

This is a nonsense response.

A bank exists not as an isolated entitity, but as a node in a local, regional and global network of transactions.

Your phone as a “classic banking infrastructure” (nonsense phrase) can’t do credit card acquiring or realtime transactions because it’s not connected to the payment rails, transaction switches and so on (like SWIFT but all of the other less global ones run by central banks and private entities).

In developed societies, instant settlement for bank-bank transfers is the norm, and cash flow is dependent on that.

Russia and Iran pay about 2-5% for above-board (non-sanctioned) cross border transactions due to their extra costs of not being in SWIFT and USD sanctions, and between 20-50% where physical middle-people are needed to move pallets of USD.

This is absolutely peak "confidently incorrect"; it's hilarious, but completely expected on this site.

It depends on the accepted downtime.

Reply on news.ycombinator.com