Ask HN: How do you deal with Fuzzy Duplicates?

Hi HN,

I would like some advice on how you guys would deal with fuzzy duplicates in a mailing list (currently an excel file with 40k+ rows) with records that have not been entered in a uniform manner. For example, sometimes just the initially be given for a name, or the physical address may be formed slightly differently, and other small errors.

I am looking to create a web service for the nonprofit that I am helping out in India to help them deduplicate files in a 'user friendly manner'. I'm not so worried about exposing it as a web service, as lost on what tools I could use. I would desperately like to use Python and Unix command line tools if possible but am not sure where to start.

Thanks