Hacker News Reader

Comment by 1amzave | Hacker News Reader

I think a reasonable argument could be made that if it's in the filesystem it's not end-to-end.

Mind explaining a little more what you mean?

ZFS checksumming has served me well..

If you mean "it's not checksumming data not represented in the filesystem, like unpartitioned sectors", well, no one cares about that data, right?

See other reply: https://news.ycombinator.com/item?id=10363281

end-to-end checksumming refers to the fact that when a file is written it's checksum is generated and stored, upon reading the file system verifies the checksum before returning the data to the application. That is, the read() will return valid data.

See this post from Jeff Bonwick: https://blogs.oracle.com/bonwick/entry/zfs_end_to_end_data

Realistically it's not going to be that simple though. The filesystem proper (e.g. ZFS) may verify checksums when pulling in data from a storage device, but then it's going to sit in the page cache for some arbitrarily long time. Whether by faulty DRAM, cosmic rays, or writes through stray pointers, that's a potentially major source of vulnerability to data corruption. There's of course a mirror-image vulnerability on the write path: you write into the page cache, and if something borks your data between that point and when it actually gets written back to storage, you end up persisting it (quite arguably worse, if perhaps less likely due to dirty data usually getting flushed within a reasonably short time).

Basically, the filesystem isn't the end point, and thus simply isn't positioned to really provide "end-to-end" protection.

If your OS allows writes through stray pointers, you have bigger issues like code exec...

Also, ZFS only has it's claims to data integrity if you use ECC memory.

BTW, anything running on top of the machine is going to be vulnerable to your example, i.e. if you have a program that checksums something, and then writes it to disk, there is opportunity for corruption while it's reading the data to checksum it.

Reply on news.ycombinator.com