Comment by hn_throwaway_99 | Hacker News Reader

Comment by hn_throwaway_99 | Hacker News Reader

Parent Full thread

hn_throwaway_99

This article touches on a lot of different topics and is a bit hard for me to get a single coherent takeaway, but the things I'd point out:

1. The article ends with a quote from the Backblaze CTO, "And thus that the moral of the story was 'design for failure and buy the cheapest components you can'". That absolutely makes sense for large enterprises (especially enterprises whose entire business is around providing data storage) that have employees and systems that constantly monitor the health of their storage.

2. I think that absolutely does not make sense for individuals or small companies, who want to write their data somewhere and ensure that it will be there in many years when they might want it without constant monitoring. Personally, I have a lot of video that I want to archive (multiple terabytes). I've found the easiest thing that I'm most comfortable with the risk is (a) for backup, I just store on relatively cheap external 20TB Western Digital hard drives, and (b) for archival storage I write to M-DISC Bluerays, which claim to have lifetimes of 1000 years.

I personally don't believe an archival storage, at least for personal use.

Data has to be living if it is to be kept alive, so keeping the data within reach, moving it to new media over time and keeping redundant copies seems like the best way to me.

Once things are put away, I fear the chances of recovering that data steadily reduce over time.

> Once things are put away, I fear the chances of recovering that data steadily reduce over time.

I’ve run into this a lot. You store a backup of some device without really thinking of it, then over time the backup gets migrated to another drive but the device it ran on is lost and can’t be replaced. I remember reading a post years ago where someone commented that you don’t need a better storage solution, you need fewer files in simpler formats. I never took his advice, but I think he might have been right.

I won't archive anything on portable media.

Cloud, or variants thereof, is fine -- I use rsync.net for backup and archive. But needing to manually run a backup (say, onto a thumb drive) is not sustainable, and even though the author suggests that disks (spinning rust or optical) might actually have a reasonable lifespan, I don't trust myself to be able to recover data from them if I want it.

As the author says, the limiting factor isn't technical. For media, it's economic. For any archival system it's also going to be social. There's a reason that organisations that really need to keep their archives have professional archivists, and it's not because it's easy :).

Only 'online' data is live/surviving data... So I keep a raid5 array of (currently 4) disks running for my storage needs. This array has been migrated over the years from 4x1 TB, to 2TB, to 4TB, 8TB and now 4x 16TB disks. The raid array is tested monthly (automated). I do make (occasional, manual) offline backups to external HDD's ( a stack of 4/5 TB seagate 2.5" externals), but this is mostly to protect myself from accidental deletions, and not against bitrot/failing drives.

Tapes are way to slow/expensive for this (low) scale, optical drives are way to limited in capacity, topping out at 25/50GB, and then way to expensive to scale.

You don't need constant monitoring if you have extra disks. If your budget is at least a thousand dollars, you can set up 4 data disks and 4 parity disks and you'll be able to survive a ton of failure. That's easily inside small company range.

My takeaway is that for personal/SMB use you have to use the cloud.

You don’t have to (though it can make sense, but I’d encrypt everything and have an additional backup). Another fairly straightforward solution (in that there is ready-made hard- and software for it and doesn’t need much maintenance) is to use a NAS with RAID 5/6, and to have a second NAS at another location (can be a friend or relative — and it can be their NAS) that you auto-backup the first one to over the internet.

Reply on news.ycombinator.com