Comment by klodolph | Hacker News Reader

Comment by klodolph | Hacker News Reader

Parent Full thread

> The risk that you lose both your primary data, (optionally your local backup) and your online backup at the same time is pretty insignificant, given that the online backup will be uncorrelated as in a different physical location (primary and local backup very correlated I agree).

Actually, that reasoning is exactly what I wanted to talk about. When the primary is lost, there is a much higher chance than you'd expect that a secondary contains unrecoverable errors. This is why RAID 5 arrays fail to rebuild after a disk failure so often—they're supposed to be able to tolerate a single disk failing, but they can't tolerate a disk failing and any other IO error at the same time. Part of this is due to how short the timeout is for failed reads in RAID setups, but I've still seen a lot of RAID 5 arrays fail, and I've seen a few RAID 6 arrays fail too.

On top of that, there's the high chance of configuration errors in DIY systems.

Agree. Having a script doing a full read of all the data every couple of months and sending a report by email (that you would notice if not sent) is a sort of must have.

I am still looking for a good way of doing this.

Weekly integrity check of all backed up data. E-Mail which informs of result. Web interface which shows overview of results of historical checks. External service which sends E-Mail if integrity check failed to run (e.g. https://deadmanssnitch.com/).

FreeNAS gets fairly close to this out of the box. My home server runs ZFS with RAIDZ2. By default, I think, there's a weekly cronjob to scrub the ZFS pool (integrity check everything, as I understand it), and then the results of that scrub are emailed to me.

I don't believe it has a web interface with historical checks, although I could be wrong. That said, it might be stored in a log file somewhere.

I also don't have an external service that would send me an email if it failed to run. That said, I would get an email if cronjobs had a mysterious error; otherwise, if the server itself was dead, my data would not be accessible on my home network, so I'd notice.

If the home server dies tragically, well, I hope Google Cloud Storage is doing similar integrity checks -- that's where my offsite backups are.

Integrity is harder because you have to sort of maintain a signature of every file on the side. What I do is to just have a script that reads all data. If some data is unreadable an exception will be thrown and a text message will be sent to me. And a confirmation email is sent otherwise.

In .net it's only a few lines of codes. Haven't thought of deadmansnitch but it's a good idea. Would just take one more line of code.

git-annex has fsck command that can test the data and rclone remote so you can use any configuration of "cloud" storages to hold the chunks.

Reply on news.ycombinator.com