Ask HN: Why aren't tweets picked up by the duplicate filter? | Hacker News Reader

Ask HN: Why aren't tweets picked up by the duplicate filter?

13 pointsanigbrowl11 comments

I'm not sure Elon Musk shitposts constitute news at any time, but why are people able to post the same tweet over and over without being flagged by HN's otherwise-efficient duplicate filter? HN strips query data from URLs before they are posted, maybe it should do so prior to the duplicate check too since most Twitter URLs come with a large helping of cruft.

And please, check the New page before submitting. There are 4 links to the same evidence-free tweet in the last 10 minutes.

View on HN

Possibly related - looks like recent posts are missing in Algolia search index:

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

There's a button for that and you have enough karma to use it.

But really I can't decide if links to Twitter are better or worse than links to Medium. 1 time in 20 or so somebody posts a highly intelligent series of tweets that are worth reading, but most of the time a link to twitter points to a poster that doesn't really have anything to say, a bunch of comments that say at best "me too!" and a lot of irrelevant junk. (Almost as bad a link to a "good" discussion on reddit that is spammed up with irrelevant discussions like a porn site where you have the hardest time telling your wife, 'ignore the 15 animated gifs of pistoning action and look at that picture which I think is really hot')

What's the button, and what's the karma threshold?

Flagging posts/comments you dislike. I think the threshold is 2000 or something but I haven't thought about it in years.

Unfortunately the comment you replied to misunderstood my question. I wasn't complaining about tweets in general (though I find them mildly annoying) but expressing confusion about why a specific tweet (with an inherently unique URL) was being posted over and over when the HN dupe filter should have prevented that.

It's not intended to mean "dislike" but rather that you believe it is a violation of the guidelines. It's part of the system for keeping the front page clean.

Anybody who is posting anything about the twitterpocalypse however is pretty oblivious because if they were reading the front page they'd notice that nothing about it is making the front page, including the people who posted an "Ask HN: Can you knock it off posting about Twitter?"

The question was about the duplication filter specifically, not about twitter in general or the twitterocalypse or why HN users post tweets.

My question about buttons was about this also. Is there a dupe button? How do posts get the [dupe] label if not? Is it manually applied after people generically flag it?

Normally if you submit the same link as someone else it will be automatically flagged as a duplicate, and you'll get a link to the original post. This works well most of the time, which is why I was surprised that individual tweets (with their very specific URLs) didn't set it off.

The duplication filter is run by HN rather than user flags. Most of the time it's effective, although if someone posts an interesting story at the wrong time of day and it dies for lack of attention the dupe filter can defeat subsequent efforts by people to repost it. However it's easy to circumvent by modifying the submission URL, just don't abuse it.

See https://github.com/minimaxir/hacker-news-undocumented#flaggi...

Dang's reply: https://news.ycombinator.com/item?id=33790811

>they've found the problem and are currently backfilling the missing data

I have no affiliation with HN.

The duplicate filter, along with all of HN code is a work in constant review, rewrite, test, and publish.

Setting the parameters for such filtering requires time, https://news.ycombinator.com/from?site=twitter.com/elonmusk

to adjust window of operation, frequency, etc. https://news.ycombinator.com/from?site=twitter.com

Reply on news.ycombinator.com

Ask HN: Why aren&#x27;t tweets picked up by the duplicate filter?

Ask HN: Why aren't tweets picked up by the duplicate filter?