If you don't want on site, a second off site provider, 50TB on backblaze would be an extra ~$350 per month (prob less if they were to commit to capacity instead of using their PAYG service)
I wasn't 100% sure, but the data backs this up[0]; the "3-2-1 backup rule" is a relatively modern concept, at least by that name. I also remember things like "have one backup in one place, another in another place" since before this mnemonic, but I feel like it's unfair to judge a 70+ years archive on something that just recently actually became a widespread mantra.
- [0] Interestingly, Ngram (https://books.google.com/ngrams/graph?content=%5B3-2-1+backu...) says ~2010 while Trends (https://trends.google.com/explore?q=3-2-1%2520backup%2520rul...) says slow build up from ~2020, guess "everyday" language would sit somewhere around those two different truths.
While statistically it might equate to "unlikely" as you state, the fact it did happen, I now have that experience that is not forgotten. Never bring a large storage pool into PROD immediately after the array finishes striping. Put it under load for a period of time first. The question remains on how long that period is though which is probably answered with "how paranoid are you"
While there's always luck involved, dying during warmup and not two weeks later is the expected experience for drives that defective.
So what happened to you wasn't that you made a backup and then it failed so fast you couldn't use it. It's relevant to what I was saying is unlikely, but it's not an example of it.
Plus at zero weeks or even two weeks you have the previous backup system around. If you have a big simultaneous failure 6 months out, that's a scary situation, but that's much rarer.
I mean, that 70+ year old archive must have been digitised at some point, unless Iron Mountain was storing the reel to reel tapes and/or film from the original recordings. So some "modernation" has been applied to it since its creation.
Now add in tons of other legacy formats and digitising becomes a pretty big cost for a public station.
They signed up with them in 2019. Not long ago.
Even if it is a "new" concept, common sense isn't. Pretty much anyone who cares about their data will do the due diligence, and this is not something that requires technical chops. This has been a concern for well over 20 years!
It really sounds like some senior management trying to be cheap.
They probably have 1-2 IT staff, who have been paid significantly under market value compared to private sector.
Management was not management of an IT organization, and probably did not really understand the trade-offs or what they'd lose by not spending more money on IT. Their IT staff was probably super busy and over-worked and also (not to throw them under the bus) not experienced enough to really advise them, and if they had and had tried to advise them, would management have decided that doing less programmatic work to pay for it, and having IT people at least temporarily spend less time on whatever they do to support regular business, to set up and support backups, was worth it? Maybe, maybe not. Again, the managers probably have little experience with IT. The whole organization is probably well under 100 people.
Source: I work in non-profits.
Why is HN always focused on suggesting there's a person who could have easily made the right choice if they weren't an idiot?
I'm familiar with the dynamics of such places.
If they actually go through a proper lawsuit, my guess is the lawyers will cost more than whatever they'd have to pay for redundant backups by an order of magnitude. Which means they probably won't go to court.
I don't know why they didn't decide to do that, I guess they just make poor decisions to be cheap. Otherwise it's a complete mystery.
Also, this TV station has not been sending their show archives to Iron Mountain for the past 70 years. Probably what happened was: In 2019, they had an on-site or nearby archive. Then some enterprising IT director came along and convinced management that they would save a bunch of money and time by outsourcing everything to a third party who said they could do it for cheap. Very confident in their genius, they then scrapped their own archive entirely.
Part of me pities them for ending up in this situation, the other part of me believes they got everything they asked for.
Which is not as thorough, but still makes the point that you should do more than one backup
Probably! The important part is that your pretend story time supports your base desire to blame the victim!
I've heard variations of this all the way back to some 1990s vocational IT courses in the country I live, and it wasn't even technologically advanced at the time. It's hard to have any sympathy for someone who misses this concept in today's age.
Non-profit IT has always seemed, to me at least, to be a land of compromises. There's rarely enough money, knowledge, or time to do things "the right way", so you have to do the best you can. If I could only choose one backup to have, it'd be remote on a provider like Iron Mountain. It would seem as if they are storing things on Iron Mountain through an intermediary company, who may or may not have actually paid the bill.
So, y'know, fraud's a possibility here.
In a free society, we don't need state-run media.
The US is not a "free society".
Which is why the BBC, ABC, CBC, etc exist in civilized societies.
I think this is a kind of time-sequence fallacy which I've seen before on HN. It's not that "we don't," but that, perhaps, "we wouldn't."
E.g., I study the pieces still standing in famous checkmate positions and decide to start the game using only those pieces. Bad strategy.
Sounds like they paid a company to digitise all their old tapes, and store the results. They possibly still have all the tapes (on site) and are just trying to avoid paying the costs to redigitise them.
But, it’s also quite likely they destroyed all original the tapes after digitisation, and this was the only copy.
https://www.ibm.com/history/magnetic-tape
Usually what kills the tape is the little reflective stickers on the back side of the tape material will fall off and render the tape useless since the machinery can't tell where the recording starts. Apparently I bought cheap mag tapes back then.
At 50TB, they probably could have Just™ written three copies of the data to a pile of disks, sent one copy off to Iron Mountain, and kept the other two copies unplugged in a safe. I can understand the DIY solution might not be allowed for insurance reasons, but it's something so cheap they probably could have done both. Unless OSS was just taking them for a ride on storage, which... well, actually that seems pretty likely.
[0] To put it simply, there is no good medium for the archival of digital information. HDDs are not shelf-stable. Flash is even less shelf-stable than disk. Sony is actively trying to kill optical. LTO is shelf-stable, but good luck buying known-good or new drives for old formats. The only option is periodic cloning and verification of offline disks, or keeping all your disks online, which is the NAS solution.
IM seem to be holding KETC's data hostage, which is Not Cool.
Maybe they mixed data from many companies on the same volume like S3 does. Maybe there were ACLs on that. Maybe it includes PII. Maybe it's a bunch of shards and needs an index on OSS servers to reassemble. Maybe it's encrypted by keys OSS had.
2. That doesn't actually seem to be Iron Mountain's argument here, though the reporting isn't crystal on the point.
There is no way for Iron Mountain to determine who OSS’s customers are, and their permissions, without help from OSS itself.
And because OSS is defunct, they can’t get that help.
How does AWS then determine which files are whose when my angry clients go knocking on their doors because I've decided to disappear?
2 it doesn't say either way. From owning systems like this, I'd assume there's some degree of "we have no idea what's in this bucket and can't just hand it over to random person asking for it".
Context: I owned a photo backup startup. 8.6 billion photos. Some might be yours, most are not. If you went to AWS and asked for a copy of the bucket, they rightfully wouldn't have complied.