HN
Hacker News
Top
New
Best
Ask
Show
Jobs
Comment by jimsimmons | Hacker News Reader
Parent
Full thread
jimsimmons
·
What is the best way to deduplicate a corpus of documents
View on HN
marginalia_nu
·
If you mean in the sense of dealing with documents that are very similar but not binary identical, a locality sensitive hash would do the job.
OnlyMortal
·
A Rabin finger printing algorithm and a hash of the data it generates.
Reference count the hashes.
Reply on news.ycombinator.com