Hacker News Reader

Comment by OroPla | Hacker News Reader

Bit rot was mentioned in the article, but then no solution was offered. I also haven't found a good solution, unfortunately. If you only have a couple of Terabytes you could create check sums of everything and then verify regularly, but I feel this approach doesn't scale very well if you have lots of data.

View on HN

If you have lots of data, I would suggest using a filesystem designed for this, like ZFS or BTRFS. You would still have to spend in storage though, as ultimately protecting from bit rot requires redundancy.

I do use btrfs, but I don't really understand how to use it to protect against bit rot. I tried to look into it at some point, but got overwhelmed. Did invest in two backup copies, one of them off site, though.

However, looking at prices, I can see one of those backups just being dismantled to be used as more storage.

Zfs and btrfs will avoid corruption caused below the file system layer, not something like malware or misuse from a tired sysadmin. I run a tool called integrit that creates checksums and lets me detect non-wanted changes. I only run it on important stuff that does not change too much like photos and videos. Not fun to have to regularly check the log but it works fairly well. This could maybe be extended with automatic checking of the logs.

If you got overwhelmed in trying to understand how to do it, just do it the easy way: create a ZFS pool of two drives of identical size in mirroring. Then, make sure to have regular scrubs (this means ZFS will regularly check all files for corruption, and recover from eventual errors), use your OS' task scheduler (e.g. Systemd or cron) to run this once a month. This should be sufficient to protect you from bit rot.

I've currently got two 10TB, one 14TB and one 18TB drive. I don't think what you describe is as easy for me as you expect. I've got one backup in a different city and one in cold storage.

The sizes are all over the place because I only buy a new drive when I run out of storage and then just get the largest one within a reasonable cost. It has been cost prohibitive to mirror everything and current prices aren't encouraging. Neither is the rate at which HDDs get larger. I was kinda hoping for 100TB drives under $400 by 2025 back in 2014, but that hope has been shattered.

Sorry for the late answer. Are all these disks full with data? Otherwise I would start with a pool of two 10TB HDDs in mirroring.

Reply on news.ycombinator.com