Hacker News Reader

Comment by embedding-shape | Hacker News Reader

> but what ever happened to the 3-2-1 backup rule?

I wasn't 100% sure, but the data backs this up[0]; the "3-2-1 backup rule" is a relatively modern concept, at least by that name. I also remember things like "have one backup in one place, another in another place" since before this mnemonic, but I feel like it's unfair to judge a 70+ years archive on something that just recently actually became a widespread mantra.

- [0] Interestingly, Ngram (https://books.google.com/ngrams/graph?content=%5B3-2-1+backu...) says ~2010 while Trends (https://trends.google.com/explore?q=3-2-1%2520backup%2520rul...) says slow build up from ~2020, guess "everyday" language would sit somewhere around those two different truths.

View on HN

I can pretty clearly remember hearing about the "3-2-1 backup rule" by name 10 or 15 years ago on the Security Now podcast with Steve Gibson. So it is _at least_ that old, but probably even older. The concept of having multiple backups is not at all new, however. Since I've been involved with computers (~30 years), I have heard folks say things like "two is one, one is none" in reference to backups and data replication in general.

Also, this TV station has not been sending their show archives to Iron Mountain for the past 70 years. Probably what happened was: In 2019, they had an on-site or nearby archive. Then some enterprising IT director came along and convinced management that they would save a bunch of money and time by outsourcing everything to a third party who said they could do it for cheap. Very confident in their genius, they then scrapped their own archive entirely.

Part of me pities them for ending up in this situation, the other part of me believes they got everything they asked for.

The rule I always remember hearing is “however many backups you think you have, you have one less”

Which is not as thorough, but still makes the point that you should do more than one backup

Two is one, and one is none.

> Probably what happened was:

Probably! The important part is that your pretend story time supports your base desire to blame the victim!

The concept of multiple copies in multiple locations already existed when backups on CD-R were common, so 20+ years ago. The “2” in the 3-2-1 rule refers to using different physical storage media and predates cloud storage.

This has been a rule for like 30+ years (I have only been in the industry that long). Whether it was called "3-2-1" before that or not, the practice is the same. Have an onsite backup, have an offsite backup. This isn't a new concept at all.

Your description leaves the reader wanting. You make it sound like having a backup on an HDD in one location and then having another copy on the exact same make/model HDD in another location is a-okay. What if the vendor had a defect that strike the make/model of HDD used?

That's pretty unlikely, but even if it does happen to be a 3TB Seagate or something you'll be pretty safe. The offsite backup is probably offline so no drives are failing, the onsite backup is still pretty likely to survive, and you have the originals.

You say that to someone that has personally brought up 4 16-drive 4RU units on a Friday with brand new drives and then left over the weekend to complete the striping and an initial warm up process of writing/deleting files to the array. When I arrive on Monday, multiple drives in each enclosure had failed which meant not data recovery would have been possible. Luckily, it was during the warm up and not 2 weeks later after being active. After contacting our vendor, all drives were replaced not just those that failed. Later discovered that it was a manufacturer issue, and multiple other clients of our vendor had similar experiences.

While statistically it might equate to "unlikely" as you state, the fact it did happen, I now have that experience that is not forgotten. Never bring a large storage pool into PROD immediately after the array finishes striping. Put it under load for a period of time first. The question remains on how long that period is though which is probably answered with "how paranoid are you"

> Luckily, it was during the warm up and not 2 weeks later after being active.

While there's always luck involved, dying during warmup and not two weeks later is the expected experience for drives that defective.

So what happened to you wasn't that you made a backup and then it failed so fast you couldn't use it. It's relevant to what I was saying is unlikely, but it's not an example of it.

Plus at zero weeks or even two weeks you have the previous backup system around. If you have a big simultaneous failure 6 months out, that's a scary situation, but that's much rarer.

> the "3-2-1 backup rule" is a relatively modern concept, at least by that name. I also remember things like "have one backup in one place, another in another place" since before this mnemonic, but I feel like it's unfair to judge a 70+ years archive on something that just recently actually became a widespread mantra.

They signed up with them in 2019. Not long ago.

Even if it is a "new" concept, common sense isn't. Pretty much anyone who cares about their data will do the due diligence, and this is not something that requires technical chops. This has been a concern for well over 20 years!

It really sounds like some senior management trying to be cheap.

It is definitely not having sufficient budget, although I doubt management was "trying" to be cheap.

They probably have 1-2 IT staff, who have been paid significantly under market value compared to private sector.

Management was not management of an IT organization, and probably did not really understand the trade-offs or what they'd lose by not spending more money on IT. Their IT staff was probably super busy and over-worked and also (not to throw them under the bus) not experienced enough to really advise them, and if they had and had tried to advise them, would management have decided that doing less programmatic work to pay for it, and having IT people at least temporarily spend less time on whatever they do to support regular business, to set up and support backups, was worth it? Maybe, maybe not. Again, the managers probably have little experience with IT. The whole organization is probably well under 100 people.

Source: I work in non-profits.

Why is HN always focused on suggesting there's a person who could have easily made the right choice if they weren't an idiot?

You say management was not being cheap, and then list everything cheap management does.

I'm familiar with the dynamics of such places.

If they actually go through a proper lawsuit, my guess is the lawyers will cost more than whatever they'd have to pay for redundant backups by an order of magnitude. Which means they probably won't go to court.

You're right, management should have "decided" to triple their budget in order to have double staff, have management with IT experience, have triple the IT staff and pay them double, and pay more for IT vendors/equipment as well, while keeping programming the same. Or heck, they should have increased programming too, why not. Maybe quadruple the budget.

I don't know why they didn't decide to do that, I guess they just make poor decisions to be cheap. Otherwise it's a complete mystery.

> I also remember things like "have one backup in one place, another in another place" since before this mnemonic

I've heard variations of this all the way back to some 1990s vocational IT courses in the country I live, and it wasn't even technologically advanced at the time. It's hard to have any sympathy for someone who misses this concept in today's age.

I'm under the impression that "two is one and one is none" is old military slang.

Redundant backups arranged to survive things like "building burns down" or a natural disaster predate digital backups.

Maybe the catchy name, but the practice was very much there in the 1980's.

The 3-2-1 rule is ~20years old at this point. But just because an archive is "old" doesn't mean we can't apply modern standards to that archive as time marches on, otherwise the BM_ARC would have never been build and opened a couple of years ago.

I mean, that 70+ year old archive must have been digitised at some point, unless Iron Mountain was storing the reel to reel tapes and/or film from the original recordings. So some "modernation" has been applied to it since its creation.

Wouldn't be surprised if it wasn't digitised. TV was using BetacamXR up until very recently (and still might be). Tape is pretty reliable and lasts awhile.

Now add in tons of other legacy formats and digitising becomes a pretty big cost for a public station.

Reply on news.ycombinator.com