Hacker News Reader

Comment by RobinL | Hacker News Reader

Naive Bayes is also surprisingly powerful for deduplication of large datasets of messy data (and the related problem of record linkage).

It's still competitive with cutting edge approaches if carefully applied, and usually much faster so it can be applied to big data.

It's the engine behind Splink, the free Python library that I develop: https://github.com/moj-analytical-services/splink

View on HN

> Naive Bayes is also surprisingly powerful for deduplication of large datasets

Thanks for this! I implemented Entity Resolution from scratch 2 years back and was mostly able to do deterministic linkages using identifiers or using TF-IDF on names as a suggestion for human intervention. Will be writing about my learnings in upcoming months [1].

Was mostly looking at research papers before but will take a look at your project - your documentation looks good!

[1] https://www.sheshbabu.com/posts/entity-resolution-challenges...

This is a very cool tool. I was testing fastlink recently but found some issues with the docs and implementation. I will check out your tool.

Congrats on winning all those awards.

Thanks, appreciate it.

Fastlink was the inspiration for Splink - the fundamental statistical model is very similar. The first version of Splink was essentially a port to make it work faster and at greater scale, but we've subsequently added quite a bit of additional functionality

Feel free to ask questions if you run into any issues - we're usually fairly good at responding: https://github.com/moj-analytical-services/splink/discussion...

I had a friend that had a record linkage problem which interested me and so I dove down a rabbit-hole of Markov logic, but didn't see any mention of Naive Bayes approach when doing so. Although I don't really have any expertise in the area anyway, so it would be easier for me to miss.

The model is usually called the Fellegi Sunter model, but once you get into the maths, it's actually the same as Naïve Bayes. If you're interested I've got a blog post that explains this here: https://www.robinlinacre.com/maths_of_fellegi_sunter/

Thanks for this, been meaning to read up on the Fellegi Sunter paper for a long time!

Reply on news.ycombinator.com