Unreliable memory can corrupt your data in ways you won't notice immediately by which point the corruption could have spread. Lets use a SQLite database as example: an index is read into memory, a bit flips in memory, the wrong result is returned by the corrupted index, the corrupted cached index is used in queries, a corrupted query result used to update the database, the corrupted cache wasn't marked dirty so it doesn't get written back to the database file. Repeatedly backing up the ticking logic bomb created this way doesn't help you.
I had backups on an external drive that I'd periodically copy data to.
I can't remember the exact sizes but this will still explain in principle what happened.
I had a 1TB drive in my desktop. I had a 500GB external drive. At time of purchase I had less than 500GB to back up.
At some point in time my desktop hard drive started corrupting data unbeknownst to me.
The amount I needed to backup grew beyond 500GB so I purchased a new larger backup drive. I did a full copy (corruption and all) from my desktop to new backup drive.
At some point I repurposed the old backup drive for something else erasing it. It is at this point I have irrecoverable data loss and I still don't know.
The corruption became so widespread on my desktop drive I became aware of it. I check my backup and discover a non trivial amount of my data was corrupted.
At first it corrupted a few files. I though nothing of it since I had a few power outages. Then more files. So I reformatted but file corruption kept happening. Switched the drive to a separate chipset with the same cable and all was good.
My current solution to this situation is a Low power PC which runs FreeBSD that has ECC RAM and a ZFS pool consisting of five mirrored drives. This PC gets backups pushed to it from my main workstation and makes a snapshot each time. I plan to change it though to a pull configuration. This way it will be immune to crptolocker software performing privilege escalation attacks since no services will be offered and no credentials will be viewed by the workstation. I have to configure it using its own keyboard though.
Even then the backups need to be tested.
Isn't that the role of zfs scrub?
Or do you mean testing if say a JPG file is still a valid JPG?
I think there are scripts that can store a md5 of each file in a sqlite database for filesystems without checksumming such as xfs
Also from my understanding TCP/IP error correction isn't that that great: https://news.ycombinator.com/item?id=25335936
It's definitely possible to write a script that compares a file across multiple snapshots and flags it if it's content changes but its modification time does not. It will just get tricky when the file gets modified between backups as the file could have been modified then corrupted then backed up. In that case how does the script know that the file has been corrupted?
It's all well if you notice it soon enough, but for rarely touched files they can drop off retention and you're left with corrupted copy
This seems like an unavoidable issue though when using a workstation without ECC RAM and a copy-on-write filesystem. I thought about moving the files off my workstation to my NAS which stores my media files. This does tick both the CoW and ECC boxes but it's not properly set up yet. Setting up an iscsi target on the NAS is an option but then it gets fidley when trying recover specific files from different points in time since I can't just browse the snapshot like any other filesystem.
That said, one of the prerequisites for this is that the stuff you're writing to the disk in the first place isn't corrupt.