Hacker News Reader

Comment by dark-star | Hacker News Reader

But can it do dedupe on such a box? I think the recommendation is still "1GB of RAM for each TB of storage" if you're using dedupe...

I still have some boards with ~512mb RAM lying around (an UltraSPARC for example) that I'd love to re-purpose to a cheap NAS, just for the heck of doing it on a non-x86 platform....

View on HN

I’ve tried dedup out, and even with a large powerful box with a LOT of duplicate files (multi-TB repositories of media files which get duplicated several times due to coarse snapshotting from other less fancy systems), I get near zero deduplication. I think it was literally low single digits percents.

ZFS dedup is block based, and actual block size varies depending on data feed rate for most workloads (zfs queues up async writes and merges them), so in practice once a file gets some non-zero block offset somewhere which happens all the time, even identical files don’t dedup.

Wow, that’s worse than I realized! Honestly, this makes me wonder whether the feature should even exist in ZFS. Given the enormous hardware requirements and minimal savings... well, I’d be curious to hear if anyone has ever found a real use case.

Does anyone actually use dedup? I think even the OpenZFS documentation says compression is more useful in practice. If at all, dedup should be an offline feature, to be run as scheduled by the operator.

My setup tries to get the absolute highest bandwidth and uses NVMe sticks in a stripe (I get my redundancy elsewhere), no compression, no dedup and yet can only hit ~ 3.5 GB/s reads (TrueNAS Core, EPYC 7443P, Samsung 980PRO, 256 GiB). I hope TrueNAS SCALE will perform better.

It would be nice if ZFS was able to combine dedup and compression - basically be able to notice that a block/file/datastream was similar/identical to another one, and do compression along with a pointer ...

ZFS can have both features enabled at once.

Though there is no clean way to disable either. Compression can be removed from files by rewriting them, but removing deduplication requires copying over all data to a fresh pool.

You should be able to remove the deduplication data by zfs send/recv within the same pool too, I think

Not according to people who know more about ZFS than me.

https://www.truenas.com/community/threads/zfs-dedup-disable-...

So at a pool level you might not be able to turn it off once it's turned on, but you can also turn off deduplication per file system, including in properties you set when receiving a stream. I wasn't confident this would work, but a test proved it can. (chicken_test/dedup_source had deduplication enabled and 16 copies of the same 100MiB file)

  chicken:~# zpool list
  NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    
  HEALTH  ALTROOT
  chicken_test    15G   144M  14.9G        -         -     0%     0%  16.00x    
  ONLINE  -
  chicken:~# zfs send chicken_test/dedup_source@send | zfs recv -o dedup=off 
  chicken_test/nodedup_dest
  chicken:~# zpool list
  NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    
  HEALTH  ALTROOT
  chicken_test    15G  2.29G  12.7G        -         -     0%    15%  16.00x    
  ONLINE  -
  chicken:~# zfs get dedup chicken_test/nodedup_dest
  NAME                       PROPERTY  VALUE          SOURCE
  chicken_test/nodedup_dest  dedup     off            local
  chicken:~# zfs destroy -r chicken_test/dedup_source
  chicken:~# zpool list
  NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    
  HEALTH  ALTROOT
  chicken_test    15G  2.29G  12.7G        -         -     0%    15%  1.00x    
  ONLINE  -

Zfs send/recv sends the blocks as written to the original filesystem (which is why it can be so fast, it doesn’t have to ‘understand’ what it happening or defragment things to read like reading a file does), but that also means undoing or applying dedup won’t work correctly unless it’s screwing with things you probably don’t want it too.

One issue I had is that due to what I eventually tracked down as power issues, I had some corrupted data written to disk under my zfs pool (at the media write later), and I had dedup on.

So dedup, unfortunately, actually made it REALLY suck to fix, because I couldn’t even copy a new version of the file to the same pool! It kept nuking the duplication, and keep the old bad data and I then couldn’t read the copy. :s

It even did this after I deleted everything, because prune couldn’t remove the bad underlying entries because it was having a media failure.

So delete files, scrub, put new files on resulted in them having the exact same failure.

When I nuked the pool and recreated it, it was all fine though.

So yeah, be careful with dedup.

Zfs send/recv actually does send data at a logical level, unless instructed otherwise. There are options to send deduplicated streams, streams maintaining compression, and raw streams but none of those are the default. Also, see my reply to a sibling comment.

practically speaking the tradeoffs to make that work are unlikely to make you or anyone else happy except in some VERY specific workloads.

My first ever large (> 4TB) ZFS pool is still stuck with dedup. It's a backup server, gets about 2x with deduplication.

At the time, it was the difference between slow and impossible: I couldn't afford another 2x of disks.

These days, the pool could fit on a portable SSD that would fit in my pocket.

Careful, file-based dedup on top of ZFS might be more effective.

Small changes to single, large files see some advantage with block based deduplication. You see this in collections disk images for virtual machines.

You might see that in database applications, depending on log structure. I don't know, I don't have that experience.

For most of us, file-based deduplication might work out better, and is almost certainly easier to understand. You can come with a mental model of what you're working with, dealing with successive collections of files.

Even though files are just another abstraction over blocks, it's an abstraction that leaks less without the deduplication.

I haven't used a combination of encryption and deduplication. That was Really Hard for ZFS to implement, and I'm not sure how meaningful such a combination is in practice.

> no compression, no dedup and yet can only hit ~ 3.5 GB/s reads (TrueNAS Core, EPYC 7443P, Samsung 980PRO, 256 GiB)

Hmmm, that 3.5GB/s sound low. From rough memory of doing initial storage benchmarking of our "new" Hetzner dedicated boxes a few months ago (AX51-NVMe, https://www.hetzner.com/dedicated-rootserver/ax51-nvme), they were giving about 10GB/s with mirrored NVMe drives.

Just logged into one of those boxes now, and it's running 2x 1TB Samsung PM9A1 drives (https://semiconductor.samsung.com/ssd/pc-ssd/pm9a1/), compression is on (lz4), and dedup is off.

(Didn't do any real tuning at the time, as these specs were already far in excess of what's needed for these servers.)

Would enabling lz4 compression be useful for your use case?

Thanks for the insight. I'll can try that. Note however that reading from the raw device on the same hardware & OS I hit 5+ GiB/s (IIRC).

> Honestly, this makes me wonder whether the feature should even exist in ZFS.

My understanding is that the OpenZFS project devs feels that the answer is an emphatic no, it should not exist, but they're committed to backwards compatibility so they won't drop it. (Take with a grain of salt; that's an old memory and I can't seem to find a source in 30s of searching.)

Wow, that is a very naive dedup algorithm.

Without restricting pretty heavily how you can interact with files or causing severe bottlenecks it's the best that can be done probably, since the FS API doesn't provide any real guarantees about what data WILL be written later, or how much of it, etc. So it has to figure things out as it goes with minimal performance impact.

Has anyone done any better at the FS layer?

While regular dedup is only a win for highly specific workload, the file-based deduplication[1][2] which is in the works seems like it can have some potential.

They discussed it, along with some options for a background-scanning dedup service (trying to find potential files to dedup), in the February leadership meeting[3].

[1]: https://openzfs.org/wiki/OpenZFS_Developer_Summit_2020_talks...

[2]: https://youtu.be/hYBgoaQC-vo

[3]: https://www.youtube.com/watch?v=hij7PGGjevc

This sounds cool. Current block dedup really seems like a simple (naive) v1 compared to what should be possible.

ZFS Dedup has been wonderful for me : dedupratio = 7.05x (144 GB stored on a 25 GB volume, and still 1.3 GB left free). I use it for backups of versions of the same folders and files slowly evolving over a long period of time ( > 15 years) that gives a lot of duplication, of course. (I could also use compression on top of it)

Backups are the 99% case for duplicate files, but aren't snapshots a better replacement in every way? Snapshots are already deduplicated as soon as you take them, plus they're instant. Maybe if your backups are coming from a non-zfs system, but you could probably convert normal backups into snapshots without too much trouble.

Why is dedup even present when the primary use case (backups) is better served in every way by snapshots?

Dedup (if it worked like it might have!) could solve the backup use case without needing to dictate your workflow.

In theory, it could also really help with virtualized disk workloads where there may be a lot of duplicated data from the base OS, but you can't use a snapshot (easily) because windows won't run from a zfs filesystem. You could maybe do snapshotting on zfs volumes, but that's not as flexible as a dedupe that worked as imagined.

Personally, I think online dedupe ends up being too expensive in memory and computation and ends up missing things because of divergent block sizes or offsets as another poster mentioned. ZFS doesn't support an offline dedupe, but I think btrfs does. That might be more interesting. It's still expensive to find duplicates, but it's possible, and it'd be neat to be able to rewrite the metadata to refer to a single copy and free some space, maybe.

Yeah, I think the author may be mixing up recommendations for dedup vs non-dedup. The solution is always to not enable dedup, it's a niche feature that's not worthwhile outside of very specific scenarios.

The speed you want them to run is a factor also. The rule of thumb hasn't applied for a while, he's right in noting that in the post.

I thought it does generally apply for dedup, though, because ZFS is then required to keep the dedup tables in memory?

It isn't required (in the strict sense) to keep the dedup table in memory, the problem is that performance is dire when it doesn't. It would be pretty similar to virtual memory thrashing, when the table is not fully in memory.

Reply on news.ycombinator.com