Comment by Alifatisk | Hacker News Reader

Comment by Alifatisk | Hacker News Reader

I have recently stumbled upon a challenge with deduping lots of media. I'm talking about both images and pictures. They are named in different ways, comes in different image and video formats, and are spread out throughout a whole directory and deeply nested. In total, it's about 170 GB of content.

My approach have been this: - Flatten out all the deeply nested folders - Categorize the content into folders with the year as name - Run deduplication and set the threshold to 100% match (I really don't want to loose any content) - Sync to Google photos

Have anyone else faced similar challenges?

You might be interested in this app: https://github.com/qarmin/czkawka

If you are only willing to drop exact duplicates, why not just take a file hash of everything as your starting point? Delete all but one of the collisions.

DupeGuru has been pretty effective solution for this. I don’t know if creating a hash for every file and then checking for collision is any better way? Sounds like it requires more labor.

  find . -type f -exec sha256sum {} \; | sort

Gets you half way there.

Clever, and I appreciate your suggestion! But doesn’t look like it’s worth replacing DupeGuru for that approach.

I think lots of people have faced this challenge. There are several tools that will find and allow you to purge exact duplicates, such as dupeguru.

I may have misunderstood but why aren't standard tools (such as rdfind or fdupes) inadequate?

It's not about them being insufficient, I just didn't know about them. Thank you!

Reply on news.ycombinator.com