What prevents, or tries to prevent, bit rot in standard CPU hard drives/flash drives? Is there some built-in forward error correction at some layer? If not, how could one add some?
You'd think there'd be a market for long life HDDs.
Edit: Yes, you could just replace old hardware with new hardware, but it creates so much waste...
It apparently uses a similar basic process as those mall kiosks that take a volumetric picture of your face and then etch it into a cube of glass?
But in this case, they just write the data to the side of a small piece of glass over the course of days and it's basically permanent for all eternity unless something breaks it.
In practice, hardly anyone uses up the entire write endurance of their SSD, especially not if they're using it to store backups, archives, or other relatively static data. Flash memory that isn't worn out has much longer retention times.
Your situation is likely happened because of bad RAM.
I don't see how that could have happened, since the image wasn't ever written to. It should have been the same file on disk as the backup, since I hadn't touched it since backing up the file, but it wasn't.
> You don't need a serious amount of space, for example ECC RAM uses one parity bit for 8 bits to fix one bit flip or report 2 bit flips
The thing about parity is that for it to be useful in a data recovery scheme, you have to know what bit got flipped. That works for RAID of course, because you usually know which disk is bad and so when you replace it you can work out the missing bits using the parity disk, but I don't think it could work for an individual hard drive, since in general you don't know which bit flipped.
And the thing about checksums is that they can tell you if your data has become corrupt, but they can't be used to fix things behind the scenes.
I've been wanting to try to understand them to see if I could implement something myself for a while now
Reed-Solomon is often used in storage because it is an optimal erasure code - e.g., I know this block is missing or corrupt, correct it.
I was initially thinking OP might be talking about something like this, but I think I would have heard if all new hard drives had it built into their firmware to store error correcting codes alongside the real data and automatically fix bit flips, since I suspect that would have pretty serious performance impacts in certain cases and people would be complaining.
The entire photography community begs to differ.
Gradual degradation of stored image data across HDDs of all vendors is very well documented, mainly because the bit rot is very easy to spot when working with visual data.
Bit flips are persistent and they occur on the media itself, not in transmission. There are plenty of theories, including one that argues that bits are flipped by stray neutrinos, but nobody knows for sure.
[1] https://arstechnica.com/information-technology/2014/01/bitro...
(edited to make more readable)
Also ZFS. Part of its purpose and design goals was countering bit rot (as well as device hostility to keeping data safe), as Sun had customers affected (even with ECC). Hence the end-to-end checksumming amongst other features.
scrub exists pretty much just for bit rot: you run scrub regularly, it goes over the disk, checks that every block checksums properly, and if they don't[0] it repairs the data using the non-corrupted copy (assuming you have one).
[0] and your dataset is replicated, note that even if you don't use a raid configuration you can mark important dataset as to-duplicate for this purpose (this is not equivalent to device redundancy, it's a feature which exists solely for bitrot / corruption protection)
This is not correct. All forms of Flash memory lose data (charge in cells) over time. Even glass window EPROMs are only rated at tens of years. Every new generation of SSDs uses smaller Cells that lose charge faster. Wear and tear accelerates this. Samsung 840 with 300TB wear will lose data when unplugged for a week. https://techreport.com/review/25681/the-ssd-endurance-experi...
Its much worse in enterprise settings: https://www.extremetech.com/computing/205382-ssds-can-lose-d...
There's par2 (and its offsprings MultiPar and QuickPar), there's rsbep, there's pyFileFixity.
All have their problems why you probably won't "shield" your files with them (I don't, although I worry).