Hacker News Reader

Comment by wongarsu | Hacker News Reader

I haven't tried it myself, but the widely quoted number for old ZFS dedup is that you need 5GB of RAM for every 1TB of disk space. Considering that 1 TB of disk space currently costs about $15 and 5GB of server RAM about $25, you need a 3:1 dedupe ratio just to break even.

If your data is a good fit you might get away with 1GB per TB, but if you are out of luck the 5GB might not even be enough. That's why the article speaks of ZFS dedup having a small sweet spot that your data has to hit, and why most people don't bother

Other file systems tend to prefer offline dedupe which has more favorable economics

View on HN

That doesn't account for OpEx, though, such as power...

Assuming something reasonable like 20TB Toshiba MG10 HDDs and 64GB DDR4 ECC RAM, quick googling suggests that 1TB of disk space uses about 0.2-0.4W of power (0.2 in idle, 0.4 while writing), 5GB of RAM about 0.3-0.5W. So your break even on power is a bit earlier depending on the access pattern, but in the same ball park.

What about rack space?

Not just rack space. At a certain amount of disks you also need to get a separate server (chassis + main board + cpu + ram) to host the disks. Maybe you need that for performance reasons any way. But saving disk space and only paying for it with some ram sounds cost effective.

That works out only as long as you don’t have to replace the whole machine (motherboard & possibly CPU/CPUs) to be able to add more RAM… So essentially the same problem as with disks.

In the end it all comes down to: there are a whole lot of trade-offs that you have to take into account, and which ceilings you hit first depends entirely on everyone’s specific situation.

Why does it need so much RAM? It should only need to store the block hashes which should not need anywhere near that much RAM. Inline dedupe is pretty much standard on high-end storage arrays nowadays.

The linked blog post covers this, and the improvements made to make the new dedup better.

    (5GiB / 1TiB) * 4KiB to bits

      ((5 gibibytes) / (1 tebibyte)) × (4 kibibytes) = 160 bits

Reply on news.ycombinator.com