Hacker News Reader

Comment by adamnemecek | Hacker News Reader

I hope they add deduplication. I can’t count the number of times when I get 100 pages of results where 95 pages is from the same included library.

View on HN

(I worked on this.)

This is on our radar! We de-duplicate exact matches now, but we'd like to do the same for near-similar documents.

De-duping exact matches is a game changed -- search has been miserable to use because of the dupes for so long. I can live with near-similar documents. Very excited to test this out.

Another GitHub Code Search developer here - to add more to this, we rank all the search results, and try to bring the most relevant results to the top. Ideally, if you have 10 pages of results, you shouldn't have to leave page 1 to find what you're looking for :D

That would be a tough problem. As de-dup you probably want to show/point towards the 'original' tree. But which one is the source? Or even worse someone abandons a project but someone else forked it and kept going should it show that one instead? Or should it show the one it was forked from depending on the version number. Which one is the 'true' repo now? Most certainly an interesting problem.

I kind of don't care about correctness. Just hide results that seem to be duplicate.

I get that. Just remove the 'extra'. That is a good first pass. I was thinking the longer term you want to show the 'original' higher in the list? Wouldnt you? What sort of criteria would you use to make it so it shows one copy vs another? Probably in many cases it probably would not mater much. But if you wanted to figure out linage of imports it could be? Some projects could have thousands of forks. Yet only maybe a dozen of those actually have anything going on. Those would be more useful to show?

Reply on news.ycombinator.com