Show HN: Fuzzy deduplicate any CSV using vector embeddings
app.dedupe.it
- No manual configuration required
- Works out-of-the-box on most data types (ex. people, companies, product catalog)
Implementation details:
- Embeds records using an E5-family model
- Performs similarity search using DuckDB w/ vector similarity extension
- Does last-mile comparison and merges duplicates using Claude
Demo video: https://youtu.be/7mZ0kdwXBwM
Github repo (Apache 2.0 licensed): https://github.com/SnowPilotOrg/dedupe_it
Background story: My company has a table for tracking leads, which includes website visitors, demo form submissions, app signups, and manual entries. It’s full of duplicates. And writing formulas to merge those dupes has been a massive PITA.
I figured that an LLM could handle any data shape and give me a way to deal with tricky custom rules like “treat international subsidiaries as distinct from their parent company”.
The challenging thing was avoiding an NxN comparison matrix. The solution I came up with was first narrowing down our search space using vector embeddings + semantic similarity search, and then using a generative LLM only to compare a few nearest neighbors and merge.
Some cool attributes of this approach:
- Can work incrementally (no reprocessing the entire dataset)
- Allows processing all records in parallel
- Composes with deterministic dedupe rules
Lmk any feedback on how to make this better!