Hacker News Reader

Comment by mmooss | Hacker News Reader

I've always wondered about similar designs: Doesn't calculating a hash of every block, on every read and every write, create lots of overhead? Why isn't that a problem?

Some systems have dedicated crypto co-processors for confidentiality (encryption) - e.g., I think drives with FDE, and I think Apple Silicon SoCs might have them. Can those be repurposed for hash calculation? What about systems that lack them?

View on HN

Both ZFS and modern btrfs support a large set of checksums.

Both implement sha256, which does impose a heavy speed penalty.

ZFS allows you to adjust the checksum on the fly, using something faster (Fletcher) if desired.

In btrfs, a global checksum is set at filesystem creation; xxhash is the best modern option.

There is a website: https://xxhash.com

Deduplication adds concerns for a strong hash free of collisions.

Fletcher has been the default for quite some time.

Specifically "fletcher4":

* https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops...

* https://openzfs.github.io/openzfs-docs/Basic%20Concepts/Data...

* https://en.wikipedia.org/wiki/Fletcher%27s_checksum

* https://people.freebsd.org/~asomers/fletcher.pdf

* https://www.intel.com/content/www/us/en/developer/articles/t...

Originally documented in the paper "An Arithmetic Checksum for Serial Transmissions" (referenced also in RFC 1146, in the context of TCP).

Yes, it adds some overhead, but it's fine IME. Granted, it helps that compression can significantly speed up performance. (I was very confused the first time I saw ZFS reading data faster than its drives were physically capable of, because it turned out the CPU could decompress faster than the drives could read)

Programs like filesystems typically have their own schedulers that aggregate writes, it would be an extreme performance hit if they wrote each checksum individually, as a separate I/O operation. They are certainly bundled with some other data that needs to be written.

And if you are concerned about the compute rather than storage, then writing to a block device is still slow enough so that computing a checksum isn't important performance-wise.

Reply on news.ycombinator.com