Hacker News Reader

So if a sweet spot exists where dedup is widely beneficial then:

Is there an easy way to analyze your dataset to find if you're in this sweet spot?

If so, is anyone working on some kind of automated partial dedup system where only portions of the filesystem are dedupped based on analysis of how beneficial it would be?

View on HN

Are there any tools that can run (even across network on another box) to analyze possible duplication at various block sizes?

I am NOT interested in finding duplicate files, but duplicate slices within all my files overall.

I can easily throw together code myself to find duplicate files.

EDIT: I guess I’m looking for a ZFS/BTRFS/other dedupe preview tool that would say “you might save this much if you used this dedupe process.”

I can't speak to the first one, but AIUI the ZFS way of thinking about the second one is to create a new filesystem and just mount it where you want it, versus "portions of the filesystem" which I doubt very seriously that ZFS allows. Bonus points that in that scenario, I would suspect the dedupe and compression would work even better since any such setup is likely to contain more homogeneous content (music, photos, etc)

Reply on news.ycombinator.com