Ask HN: Does HN need a dupe link detector? A better one? | Hacker News Reader

Ask HN: Does HN need a dupe link detector? A better one?

15 pointschanux9 comments

I was experiencing whole lot of dupes on HN lately.

Here comes a dupe, then someone comes and comment about the dupe and normally gets downmoded. But in my opinion down moding is not the answer. It's not gonna do any good. Dupe links shouldn't come on HN.

I have a few reasons for not liking dupes.

1) It might take away some good content out of light. Yes you can avoid up modding dupes but hey it's not just you and me on HN.

2) It might hurt some enthusiastic submitters. Yes there are people who love to take great news early on HN. And they are the people who provide good stuff for the great people who upmod _new submissions_. They are the spirit of HN quality apart from great commenters.

If their submssion goes stale while some others' go shining, it might discourage them. Especially people who are looking for some karma just for being able to be a full HNer.

3) New on HN: Sensational headings. The first days I came here, I learned that HN is not digg and sensational headings are not really welcome unless it adds some value (maybe like humor). But lately I learned it's not the HN I started with. Same story with a sensational heading goes up.

4) It encourages only mainstream blog/site content submission. (Did anyone notice that variety is one great thing about HN?.)

I've noticed this happening several times. Someone posts an article from a lesser known site. Nothing happens. Same news on techcrunch, wired or somewhere like that and gets upmodded. (Well this is not about dupe links but it counts).

I saw a comment suggesting to merge similar threads. Maybe after merge request from users.

Last and yes least, I feel like some old great submissions being resubmitted by karma whores. But this is just a feeling :).

PS: All these mean that HN is growing. And as I believe we are getting more an more normally distributed. And also, I didn't see an Erlang week recently.

I tackled this for a YC-funded company in the recent past, and it's fundamentally trickier than you might think.

If filtering out duplicate urls isn't good enough, matching copied content is the next easiest way to solve this, but it requires analyzing/storing/indexing relatively massive amounts of data. And even with that cost, there's the question of how much overlapping content actually makes something a "dupe". Overlapping subjects?

Crowdsourcing gets into merging comment threads and system gaming.. and the end result is still that one URL (probably from the most popular, "trusted" source) ends up on the front page anyhow.

So.. lots of messy boring work and higher server expenses, for .. ?

I'd call it good enough, myself

HN doesn't need a dup detector - it has that, humans.

It needs a better policy/idea of what to do about dups.

My suggestion:

Merge the comments, have a mod pick a title and url for the main view. Show the other/alternate titles and urls on the comments view.

The mod ranking is whichever is higher, and any future mods get credited to all submitters.

Have a flag called dup, which asks you for the id (or url) of the dup, and notifies a mod.

I see your point. But still there are times that people submit dupes just because they do not know. I think they need a little help (I'm in this category).

Yes the humans filter well. But loads of links on new submissions page drags things out of attention of humans. Dupe links only help very little in this case. But spam does. I don't know why I see so many mozilla start page submissions in new submissions page (for example).

Yes, I think a better one is needed. HN only seems to detect duplicates when a new submission has a URL that is identical to a previous submission.

The idea of pattern-matching URLs could help, but a lot of times the problems are due to links through blogs, etc. that have completely different URLs. So in addition to automatic detection, there should be a way for users to specify which HN pages belong together.

I'd say if the dupe hasn't been submitted in a long while then it should be accepted. A link should be added to the story pointing to the old discussion thread for that link.

This way if the community finds an old topic/link interesting again for some reason there can be a lively discussion with participants that may not have been around previously.

Agree. User confirmation for the submission over a dupe detection would solve this easily.

yes please: http://[www.]somedomain[:80]/somepath[/][index.html][#someanchor]

Otoh the ever shorter 'history' of links that you can see through the 'more' links makes it very easy to miss stuff.

On the new page it is 13 hours long, on the 'main' page it is only about a day.

May be a flag duplicate will help. Its easier for us to tell some duplicate post which comes as blog, different url, different site etc, compared to filters.

No, it doesn't, in my opinion.

I almost never see dupe links here, and I don't think it's a problem at all.

Reply on news.ycombinator.com