Show HN: An Open Source Tool to Combat Clickbait Links | Hacker News Reader

Show HN: An Open Source Tool to Combat Clickbait Links

links.spince.com

81 pointsgoldMIT28 comments

It assumes "a relevant (i.e. non-clickbait) link would have its text appear frequently on the actual page".

Is there empirical evidence for this claim? The examples featured in the article seem mixed (some right, some wrong).

It's easy enough to test: collect human judgement scores for a few hundred article links from several news sites (an afternoon's work) and compare to the algorithm.

It seems like it'd at least catch some. How else would you measure?

Thanks for the comment. There have been a lot of studies done using the general TF-IDF approach. You can read about it on https://en.wikipedia.org/wiki/Tf%E2%80%93idf

There is also a new version planned using a different approach.

This is more or less part of the definition of clickbait: an article with a title that misrepresents the content of the article in order to garner more views.

I'm not sure that term frequency is the right way to go about the problem of clickbait. Frequently, titles and body text is meant to match up as a form of SEO.

I'm not sure if this can be done through an extension, but an alternative way to go about this is by measuring time spent on the page. If the page has X words which takes Y minutes to read, but a certain number of people bounce before that time, the score is lowered. The more users that stay on the page for Y minutes, the higher the score.

There are a lot of assumptions in that solution, but it might be worth considering.

Thanks for the suggestion. The code is open sourced on github. Feel free to do a pull request with your approach and I'll take a look at it.

Oh this is exactly what I wanted - I hope you make a firefox addon soon.

How does your approach of looking for text frequency compared to, say, pattern-matching existing clickbait titles from a database? Can the two approaches be combined (say, by using Splice to generate a corpus, remove false positives manually, then use it to train a pattern matcher?) Not having to load the linked article has huge benefits on bandwidth, robustness, etc.

Thanks. This was built because someone posted an ASK HN a few weeks ago. They wanted to know if there was a way to stop clickbait. The text frequency approach seemed like the simplest and cleanest approach to build this in a short timeframe.

Firefox addon is in the works. If you want to contribute code, go to github and make a pull request.

So that means it's fetching the content (or at least the HTML portion) of every eligible link? That seems like potentially a lot of network traffic for what it does. Find if you have a fast connection, I suppose.

Can be even worse. I have seen some ugly corporate services using links with GET request to perform delete operations and having serious data loss caused by crawlers like this tool.

This was definitely a consideration. It was why there is a blacklist feature and why only links of a certain length and certain type are analyzed. Any suggestions can be made on github via a pull request.

Good point. There is some consideration for building an option to limit the number of links analyzed. So if you set it to 5 or 10, then the tool would stop at that limit. Feel free to open an issue on github.

Be interesting to train a model using user generated, is this click bait title yes/no. Is this something machine learning could get very good at?

There was a cool article on identifying clickbait headlines with machine learning just last month. (There was probably an HN discussion but I can't find it on my phone.) the main difference is that it operated only on the headline, not comparing it to the actual article.

https://www.linkedin.com/pulse/identifying-clickbaits-using-...

There are definitely training sets paired with Bayesian inference that attempt to do this.

You could probably get a decent result just by looking at the domain.

"Avoid clickbait links with this one weird trick!"

"Publishers hate it!"

Don't even need to use humans, I tried something easier -- http://github.com/peterldowns/clickbait-classifier

Why not train a network on ad unit headings from Outbrain and the like. That seems like it'd be pretty spot-on. These in-situ pseudo-related-articles ad units are, by their nature, click-bait. Aren't they?

Thanks for the comment. There is a new version under development that will use a different approach. Most likely a neural network.

Oh...nice try. I almost clicked on that link.

It seems to work pretty well. I can understand how this is far from perfect so far but the thing that bothers me the most is that there is no apparent caching mechanism and even though my internet connection is pretty fast it will take a good 10-20 seconds to go through all the links. Once you have determined a site to be clickbait or not maybe cache it so i don't have to wait 20 seconds every time i come back. Additionally this will save a lot of bandwidth.

Note: there is a form of writing where the headline introduces a weird or interesting concept and then the writer makes the reader wait for it. When done well, these can be some really good articles.

Not sure how to account for that. Just wanted to point out the lossiness of the algorithm.

Looks like a cool app. Thanks!

I like the idea behind using TF-IDF. Feels like a lot of unnecessary network traffic though.

example shows the bbc with a green dot and npr with a red one? O_O

BBC isn't as reliable as its reputation would have you believe ....

Not an issue at all but I am curious. Why "Spanish" has the Mexico flag? Was the spanish language modeled with that specific variant?

It's the largest Spanish-speaking population, maybe that's why?

Reply on news.ycombinator.com