Comment by staz | Hacker News Reader

Comment by staz | Hacker News Reader

this more like a tool suggestion request, because I haven't been able to find a solution in Google.

I'm looking for a more advanced duplicate files finder in Linux, specially one that can handle folders.

Most tool just return a list of duplicate files, but what I need is to also know is if whole folders are duplicate, or subset of others and have it presented in an elegant way t o resolve conflict. ofc everything is in a big folder and everything is a mess, bit of a hoarding collection of files that got accumulated over the years.

I could probably code some stupid script myself, if I ever got the time (spoiler I probably won't) but I have no idea on how to present the result elegantly. So it would be nice if such tool already existed.

https://meldmerge.org/ might be what you are looking for. But if there are tools which can find duplicated files and directories which are named differently, I'd be interested in learning more about these as well.

meld can compare two directory if they have the same structure and name for files but can't really tell if I inside my foot folder, I have a folder root/X which is also present as root/category/b/X. (except all the file in it have been prepended by "foo" somehow)

For tools which can find differently names files you have for examples: fslint (gui) or fdupes (cli)

That's true, meld is designed for comparing directories which already have similar structures.

It seems like what we're looking for is content-addressable storage [1]. The theory behind it appears to be based on Merkle trees and cryptographic signatures [2].

IPFS already has an implementation [3] of this, and there are other implementations (borgbackup, restic, zpaq) listed in that link and in the content-addressable storage article. Disclaimer: I haven't used any of these yet, just found them a few minutes ago.

[1] https://en.wikipedia.org/wiki/Content-addressable_storage

[2] https://gist.github.com/mafintosh/464bb8f1451f22c9e5c5

[3] https://discuss.ipfs.io/t/ipfs-and-file-deduplication/4674

Reply on news.ycombinator.com