Ask HN: How to de-dupe the HN front page | Hacker News Reader

Ask HN: How to de-dupe the HN front page

I've been using the HN client I wrote [0] for a couple of years now. It uses the official HN API. But I noticed that I get many duplicates every time I read it.

What's the best way to reduce duplicates? I always try to check recent posts before I post. Obviously if it has the same URL, it's a duplicate but it's not always that simple. Sometimes an article, like Elon Musk having Asperger's, is covered on many different sites and will appear 5 or 10 times.

(And yes, this is something that HN itself should try to prevent, but that's unlikely to happen anytime soon.)

[0] https://gabrielsroka.github.io/hn.html?small

View on HN

Use search. Especially keywords and sort by date. However, some sources are better than others and sometimes the time of posting also matters, so it is okay to have some duplicate submissions.

I always thought one of the lyrics in this song was the most sensible thing I'd ever heard:

https://www.youtube.com/watch?v=CKti7QixnJI

"Say something once, why say it again?"

Maybe personally I'd like to see "Elon Musk says he has Asperger's" exactly once on HN and never see it again.

That's what "personalization" means to me, but maybe it means something different.

If you don't want to see the same story twice, you need to implement your own filter. If it is titles only, keep index of last 1000 titles posted and apply something like tf-idf to it in order to find the most similar. You can peek in the articles bodies, but it requires proper scraping which is one more level of difficulty. So, if you find that two titles are quite similar (tune the value), just drop the one with less upvotes/comments.

Additional things to look up are stop words and embeddings.

Hope that helps

I helped build a search engine for patents that used a neural network embedding, I also wrote

https://ontology2.com/essays/ClassifyingHackerNewsArticles/

so I know what the data looks like. However that above article has never gotten as many votes as this article did

https://ontology2.com/essays/HackerNewsForHackers/

which catches the emotions better than a solution does.

I would retract some things from that article pair, for instance I put the anti-Apple filter in at the time and I did not need to, the site was just flooded with Apple articles at that moment in time because WWDC was a hit that year.

A lot is know about ranking algorithms where "better content sort higher" but not ones where the absolute value of the ranking score is meaningful, which is important for filtering applications where you want to pick out "new and interesting" articles and set some threshold for what interesting is.

As for consolidating topics that I think is better addressed through clustering than embeddings that were trained on somebody else's task.

What you want is a personalized recommending system with lots of serendipity. Maybe you can try feeding it what you've liked, accept large amount of recommendations, and remove those that are too close to the input.

Reply on news.ycombinator.com