Duperemove – Tools for deduping file systems | Hacker News Reader

Duperemove – Tools for deduping file systems

github.com

54 pointsanotherhue22 comments

That only seems to work on btrfs, XFS, (and maybe now or very soon) ZFS and bcachefs: "[duperemove] simply finds candidates for dedupe and submits them to the Linux kernel FIDEDUPERANGE ioctl." [1] (aka BTRFS_IOC_FILE_EXTENT_SAME), and this ioctl "performs the 'compare and share if identical'" (and locking etc.) work [2]. But on those filesystems, that is a nice feature, plus it lets the tool get away with a weak hash like murmur3.

1: http://markfasheh.github.io/duperemove/duperemove.html 2: https://manpages.debian.org/bookworm/manpages-dev/ioctl_fide...

A fallback to hardlinking would work for a lot more filesystems, though of course it would be limited to files that are completely identical rather than any identical extents / blocks within files.

More importantly, hardlinking means writes to one file would affect both, while FIDUPERANGE creates copy-on-write semantics.

Quick note : I removed murmur3 support, and replaced it with xxhash

I have recently stumbled upon a challenge with deduping lots of media. I'm talking about both images and pictures. They are named in different ways, comes in different image and video formats, and are spread out throughout a whole directory and deeply nested. In total, it's about 170 GB of content.

My approach have been this: - Flatten out all the deeply nested folders - Categorize the content into folders with the year as name - Run deduplication and set the threshold to 100% match (I really don't want to loose any content) - Sync to Google photos

Have anyone else faced similar challenges?

If you are only willing to drop exact duplicates, why not just take a file hash of everything as your starting point? Delete all but one of the collisions.

DupeGuru has been pretty effective solution for this. I don’t know if creating a hash for every file and then checking for collision is any better way? Sounds like it requires more labor.

  find . -type f -exec sha256sum {} \; | sort

Gets you half way there.

Clever, and I appreciate your suggestion! But doesn’t look like it’s worth replacing DupeGuru for that approach.

I think lots of people have faced this challenge. There are several tools that will find and allow you to purge exact duplicates, such as dupeguru.

I may have misunderstood but why aren't standard tools (such as rdfind or fdupes) inadequate?

It's not about them being insufficient, I just didn't know about them. Thank you!

You might be interested in this app: https://github.com/qarmin/czkawka

A while back I wrote a simple tool to remove or symlink duplicate files (exact matches). It's faster than fully hashing each file, maybe it's useful to some folks: https://github.com/abspoel/dedup

I’ve been processing a lot of niche audio files often same tracks but trimmed in length - jdupe [1] worked really well.

https://codeberg.org/jbruchon/jdupes

Do you have a solution for tracks that are identical in content but differ in metadata ? Some audio tools insist on making changes.

https://picard.musicbrainz.org/ potentially?

Microsoft’s File Deduplication works really well. But I think they only enable it in Windows Server because it would truly cut PC profits in half if people didn’t run out of storage all the time.

But Microsoft does not sell local storage?

Normal People do not run out of storage on their PCs. What data do you think normal people even have? Everything is streamed. My girlfriend has kept everything that has ever mattered to her for 15 years now, including two college degrees worth of class material, in the free tier of OneDrive, like a few gigs maybe.

Her largest digital asset is a gig of mp3s that are a copy of the Harry Potter audiobooks. Even that is "streamed" from my google drive. Even getting her into the hobby of "collecting steam games" hasn't put pressure on her 500gb hard drive, because she doesn't play modern AAA games.

People run out of storage on their phone from taking Video and Pictures but they don't ever copy those over to their PCs, if they even have PCs, they just buy more cloud storage space from Apple.

And common ways of filling up your storage, like big photo or music libraries, are unlikely to have many dupes.

> Normal People do not run out of storage on their PCs... My girlfriend...

Normal people run out of storage on their 128GB SSD macOS devices all the time.

Reply on news.ycombinator.com