Hacker News Reader

If this is the case, then why does bit-rot still occur?

It's not magic. If you have enough bit flips for the same data then it's not recoverable. Sometimes it could even flip and resemble correct data. It might not happen to you, but if 500 people reading this thread each have access to 50T of data, then sure bit rot can happen in some of that 25 petabytes.

Do you think that there is a need for some kind of ECC at the OS level? There are surely some applications where even 1 bit flip in 25 petabytes is bad. Amazon's 2008 site outage comes to mind: http://status.aws.amazon.com/s3-20080720.html

Yes, that's why you have erasure coding across drives either using RAID groups or using an object store. Just not at the single drive file system level where blocks going bad within a disk tend to be correlated (i.e. the drive is getting old and will die).

Also, EC at the block level would probably spread the blocks around the drive. This means any read would need to seek all over the dang place trying to reassemble files and that's a bad access pattern for spinning disks. Real bad. Like, the worst. It might even reduce the effective lifetime of the drive. So it would be not only correlated with device failure, it could precipitate it.

Maybe it would be ok on an SSD.

Reply on news.ycombinator.com