Hacker News Reader

Comment by fensipens | Hacker News Reader

If you want ZFS you need lots of expensive ECC RAM. (...) The rule of the thumb is that you typically need about 16GB of RAM to begin with ZFS and then about 1-2 GB for each additional TB of data.

Ugh.. can someone please tell me who started this ZFS-ECC-UBER-RAM fetish? Whenever I read anything about ZFS, some gollum first has to scream at me how superimportant a ridiculous amount of ram (especially ecc-ram) is. Why? Who said so? When?

View on HN

Here's my take on it:

One of the main selling points for ZFS is the end-to-end cryptographic verification of file contents as well as all metadata. Everything in ZFS is hashed, and verified during reads. Without ECC RAM, your hashes are vulnerable to corruption due to bits being erroneously flipped in RAM. If this happens with non-ECC RAM, you're never going to know, and it will cause corruption. With ECC RAM, these events are not only detectable, but also fixable.

Regarding ZFS "requiring a huge amount of RAM". Most of this comes into play if you have dedup turned on. Then you really need a lot of RAM. Without dedup enabled, and if you're OK with mediocre performance, you can get by just fine with a "normal" amount of RAM. It's well-known that filesystem performance increases with more RAM - this is not something that is unique to ZFS.

ZFS is fairly profligate with its RAM usage though. There are some good reasons for this, but ZFS will be dog slow compared to other file-systems in RAM-constrained situations, but behave competitively in non-RAM-constrained situations.

Here is a well balanced explanation why ECC-RAM. That being said, if the RAM is the weakest link it doesn't matter what underlying FS is used.

https://pthree.org/2013/12/10/zfs-administration-appendix-c-...

In my experience 16GB is only needed if you want to run Deduplication.

> "if the RAM is the weakest link it doesn't matter what underlying FS is used"

When is this ever the case, though? Even non-ECC RAM is more reliable than hard drives except when a RAM chip has died, which most ECC systems don't handle well either. Full chipkill capability is pretty rare.

As I understand the reasoning, it's that ZFS checksumming creates a risk of false positives where non-checksumming filesystems wouldn't have them. So without ECC RAM you're trading the risk of losing data to your IO chain for the risk of losing data to increased RAM activity. With ECC RAM it's an unambiguous win.

It sounds to me like there must be some situation where ZFS reads data from disk, verifies the checksum, keeps it in memory where it could get corrupted, then recomputes the checksum on the corrupted version and writes either the corrupted data or the incorrect checksum to disk but not both, so that the data on disk is now guaranteed to not match the checksum on disk.

Without something like the above sequence of events, I don't see how ECC eliminates any class of errors, rather than just reducing their probability. So what actually triggers such a chain of events?

This is the paper on ECC RAM and ZFS:

http://research.cs.wisc.edu/adsl/Publications/zfs-corruption...

Basically, ZFS can survive anything except for corruption of it's data in RAM, in which case it can corrupt on disk in ways there is no recovery from.

The size of RAM required is mainly due to in-memory caching - ZFS has always been RAM hungry and it's ARC performs much better with lots of RAM. This has generally scaled well on most non-single socket systems (ie, anything Xeon E5 and above) that can take >32GB of memory.

Scary scenarios regarding "faulty ram destroys volume" can be found in various places, e.g.,

https://pthree.org/2013/12/10/zfs-administration-appendix-c-...

However, for a refutation of this scenario see XorNot's comment (and perhaps the surrounding discussion) at

https://news.ycombinator.com/item?id=8294434

My impression is that ZFS makes the absence of ECC RAM more obvious. Discrepancies in RAM stand out when you're carefully hashing your file system.

Reply on news.ycombinator.com