Deduplication backend is the easiest part of the tool and it doesn't need any additional libraries. Just calculate embeddings and find close pairs. The complexity is everything around.
Using local models is worth considering and the tool already uses the LiteLLM wrapper, allowing it to configure different models, including local. I left this part for the user.
I also don't think the backend is the "easiest part" here, a lot of the scalability lives there, which is important for monorepos, or cases where you want to deduplicate across projects. For example, from looking at the implementation, you use exact brute-force similarity search (comparing every item to every other item) which is an O(n^2) operation. It also allocates large dense similarity blocks in memory, so memory use won’t scale well either.
Embedding calculation is also outsourced externally: it's only a simple API call with a LiteLLM wrapper.
"brute-force similarity search" and O(n^2) may sound scary, but it works fine and this is not a bottleneck. For large projects, other parts are much slower, which has room for improvement. [1] is an implementation you probably saw. It uses NumPy, spreads work across all CPU cores and there is also a split into blocks (block_size = 1000). In larger sets, all vectors are not loaded into memory at once. Where I need to be honest, I didn't measure actual memory usage. I just tested this on large repos, so I'm aware of bottlenecks.
From my perspective, this is the easiest part. Code extraction, chunking, applying boost, clustering, generating report and designing everything as a single user-friendly tool is a real challenge. Architectural and product decisions are more difficult than implementation and solving performance issues.
[1] https://github.com/rafal-qa/slopo/blob/v0.3.0/src/slopo/anal...
"In larger sets, all vectors are not loaded into memory at once": this is also not correct, at least in the implementation you shared. The similarity matrix is processed in blocks, but the embeddings themselves are loaded all at once and stacked into one NumPy matrix, hence the memory peak.
So in larger projects, more than half of the time is spent on the similarity search step, and almost all the memory is spent there as well.
To give more context, the current version is already an optimized one I considered good enough and didn't spend more time on it. In the first attempt, I used a vector database with indexes, trying to query for similar vectors. This was uselessly slow even in medium-sized repos. The brute-force NumPy solution is a significant improvement, making it faster than other calculations like clustering.
"vectors are not loaded into memory at once" is not true, I had in mind splitting computation into blocks.
One possible simple optimization is to use 16-bit floats in vector instead of 32-bit. I used this in a different project (halfvec in pgvector) without affecting results.
1536-dim embeddings from your case also can be reduced. This large vector usually doesn't give much benefit compared to smaller ones. And this is something I will compare in my own tests.