Comment by bane | Hacker News Reader

Comment by bane | Hacker News Reader

Parent Full thread

Goodness, I have over 100TB at home and it cost less than a two or three thousand dollars to put in place. That's like $25 per TB.

> The stored data amounts to 858TB (terabytes), equivalent to 449.5 billion A4 sheets.

No, the 858TB amounts to under $25k for the government of the 10th largest economy, of one of the most sophisticated countries on the planet, to put in place.

Two of those would be less than the price of a new Hyundai Grandeur car.

> “It’s daunting as eight years’ worth of work materials have completely disappeared.”

So they're clocking in at around 100TB/year or 280GB a day. It's respectable, but not crazy. It's about 12GB/hr, doable with professional, server level hardware with backup moved over dedicated fiber to an offsite location. Multiply the price 10x and you can SSD the entire thing.

Even with data sovereignty consideration demanding an entirely 100% home grown solution rather than turning to AWS or Azure, there's no excuse. But it's not like the cloud providers don't already have CSAP certification and local, in country, sovereign clouds [1] with multiple geographic locations in country [2]

South Korea is full of granite mountains, maybe its time the government converts one into an offsite, redundant backup vault?

1 - https://erp.today/south-korea-microsoft-azure-first-hypersca...

2 - https://learn.microsoft.com/en-us/azure/reliability/regions-...

~1PB of data, with ingestion at a rate of 12GB per hour, is a tiny amount of data to manage and backup properly for a developed world government. This is silly. Volume clearly should not have been a hinderance.

Backup operations are often complex and difficult - but then again it's been worked on for decades and rigorous protocols exist which can and should be adopted.

"However, due to the system’s large-capacity, low-performance storage structure, no external backups were maintained" ... "the G-Drive’s structure did not allow for external backups."

Clearly [in]competence was the single factor here.

This is what happens when you come up with all kind of reasons to do something yourself, which you are not qualified to do, rather than simply paying a vendor to do it for you.

> Backup operations are often complex and difficult

It quickly becomes much less so if you satisfy yourself with very crude methods.

Sure that would be an imperfect backup in many ways but any imperfect backup is always infinitely better than no backup at all.

The most sophisticated countries and companies are smart enough to use the least sophisticated backup methods. SK needs to backup their data to cassette tapes and tape libraries cost a bit more than that, but not much. Even if they boat their tapes over to an iron mountain in the US, I can't imagine the equipment and service fees are going to cost them more than a few hundred grand. They'll be spending more on the headcount to manage the thing.

The operational expenses of this stuff dwarfs the hardware cost. For the tape mountain, you need robots to confirm the tapes still work (mean time to detection of device failure and recovery are key for RAID durability computations). So, someone needs to constantly repair the robots or whatever.

If I was being paid to manage that data set, I’d probably find two enterprise storage vendors, and stick two copies of the data set on them, each with primary secondary backup. Enterprise flash has been under a dollar a gigabyte for over a decade, so that’s under $1.7M per copy, amortized over five years. That’s $700K per year, and one of the four copies (at 3-4 sites) could be the primary store.

(I can’t be bothered to look up current prices, but moore’s law says there have been six capacity doublings since then, and it still applies to flash and networking, so divide my estimate by 2^6 — so, ten-ish grand per year, with zero full time babysitters required).

even with dual vendors, you'd have to still put in place a backup/restore procedures (with the associated software, which may need to be custom). Then you'd need regular testing. These operational concerns will basically double the cost yearly, probably.

You'll need permanent staff to oversee this, too. This will add at another ~500k+ to your annual expenditure.

But it would not have been $25k, it would have been 1-2 million for an “enterprise grade” storage solution from Dell or a competitor. Which isn’t much compared with your granite mountain proposal, nor with the wages of 750,000 civil servants, but it’s a lot more than $25k.

The article reads like they actually have a fault-tolerant system to store their data. This is probably a data dump for whatever files they are working with that might have started out as a cobbled-together prototype that just picked up momentum and pushed beyond its limitations. Many such cases not only in government IT...

Looking at the article, my read (which could be wrong) is that the backup was in the same room as the original.

No. It says that "most systems" in this data center are backed up to separate hardware on a different floor, and then a backup is made at a physically remote location. This particular G-Drive system was not on the standard backup process - it sounds like it was much higher volume than any others, so maybe they couldn't use it. They did have a pilot going to get G-Drive backed up...it was supposed to be scaled up to the whole thing in December.

Wow, rough timing.

From what I'm seeing that pilot is about NTOPS (National Total Operation Platform System) which they use to manage everything, and since this was on the floor above the fire it made recovery a lot more complicated

There's a high chance i'm missing something though, where did you read about a G-Drive backup system?

G-Drive is the name of their bespoke file storage system that was taken down by the fire.

You can buy a 24Tb drive on sale for $240 or so.

Sometimes I wonder why I still try and save disk space :-/

Link? Am both curious and skeptical

https://pricepergig.com/us?minCapacity=24000 shows the cheapest 24TB drive is $269.99 right now, so yeah, with a sale you'll get to $240. But if you're ok with smaller drives, you can get a much better price per gig ratio

Seagate Expansion drives are in this price range and can be shucked. They're not enterprise drives meant for constant operation, the big ones are Barracudas or maybe Exos, but for homelab NAS they're very popular.

I have such a NAS for 8 years, (and a smaller netgear one from maybe 16 years ago), and have yet such a disk fail. But you can get unlucky, buying a supposedly new but "refurbished" item via amazon or the seagate store (so I hear), or have the equivalent of the "death star" HDDs, which had a ridicilously high burnout rate (we measured something like > 10% of the drives failed every week across a fairly large deployhment in the field - major bummer.

If you use such consumer drives, I strongly suggest to make occasional offsite backups of large mostly static files (movies for most people I guess), and frequent backups of more volatile directories to an offsite place, maybe encrypted in the cloud.

Only a fool would have 24Tb of data and entrust it to a single drive. Of course you buy more than one.

Of course you would stagger the offline backups. But if we are talking storing e.g. movies, the worst case scenario is really not so bad (unless you have the last extant copies of some early Dr Who episodes, then BBC would want to have a word with you)

Not trying to account for the parent's claim, but generally check diskprices.com for the latest deals on Amazon (.com, .co.uk, .de, .es, .it etc)

Newegg regularly offers sales on hard drives, which is when I buy.

Very possible with sales and especially non nas grade harddisks

Azure can only be sovereign to the USA.[1] [2]

[1]: https://www.computerweekly.com/news/366629871/Microsoft-refu... [2]: https://lcrdc.co.uk/industry-news/microsoft-admits-no-guaran...

Reply on news.ycombinator.com