This could be super useful for deduplicating large photo datasets!
It might be worth changing this link to the github, there’s a lot more info on the hash there.
It might be worth changing this link to the github, there’s a lot more info on the hash there.
That is indeed the use case mentioned in the article. A hash like this is useful for deduplicating large _file_ datasets but for deduplicating images in particular, you really want a perceptual hash. Because two images files can _look_ identical but have slightly (or wildly) different byte streams. The trade-off is that perceptual hashes are notorious for false positives in large datasets.
Perceptual hashing for photos isn't great in my opinion, for example I have many raw photos that look very similar, the only difference may be a slightly higher shutter speed, or a correction of the camera angle. I'm guessing many perceptual hashes would mark these as duplicate. Maybe that's what many want, it's not what I want personally.
According to xxHash's performance comparison page, xxHash is faster than MeowHash.