Hacker News Reader

Comment by lumost | Hacker News Reader

I think that's a failure of imagination. In the early days of search engines and web crawling companies paid subject matter experts such as librarians and other experts to manually grade and categorize web pages.

Spam pages tend to be filled with Ads and derivative content, owned by spam company domains, and do not provide unique information relative to other sources.

If we as humans can recognize a spam page, why can't the machines? At the very least Google can penalize repeated spam domain owners/domains to help reduce the problem.

View on HN

Can humans recognize spam pages though? Back in the day, such pages would be created on blogspot or other spammy domains, so there was at least a visual signal of some sort. Now the spam is on Facebook groups, Whatsapp group chats, TikTok, Twitter, Medium, Linkedin.....the list goes on.

Online misinformation would not be as destructive as it is if people could tell the difference between say a credible news site and a Macedonian troll farm-run 'news' site written in broken English.

> Now the spam is on Facebook groups, Whatsapp group chats, TikTok, Twitter, Medium, Linkedin.....the list goes on

Of that list, Medium is the only one I would ever expect/want a search engine to even consider returning results for searches on "how to buy a bicycle for under $500". If I was King, you'd need to add allow:social to any google search to get any results at all from anything remotely like a social media or messaging application. I'd include LinkedIn and Pinterest and similar sites in that exclusion. If you want that stuff, you need to ask for it.

> Can humans recognize spam pages though?

Probably not, but they can recognize valuable non-spam. It's called by many names, but "reputation" is as good a term as any.

Yes, I'm saying that "Facebook groups, Whatsapp group chats, TikTok, Twitter, Medium, Linkedin" are all low reputation.

If we as humans can recognize a spam page, why can't the machines?

I think that many of these automated aggregators have come a long way. On the better sites, it's not immediately obvious that the page is spam, until I've read some way into it and notice patterns in the language and a lack of "meat".

My dad can't seem to recognize spam pages 99% of the time...

I find the advertising to content ratio is a decent clue as well.

I don’t think there is a financial incentive to clean up the internet.

Too much money to be paid with shady practices that no one in power wants to do it.

Ah, the old "Step one: create a general purpose artificial intelligence."

No general AI required. Simply count the number of ads in the text.

For a top ten site I’d also expect a certain degree of interesting vocabulary which loops in industry jargon. I’d also expect spam sites to use similar syntax/sentence structure for all products.

Lastly if I can follow the links and get references to the final products, then I can estimate how similar the product description is to the product. I’d expect most customers looking at top tens want something more than what they get from searching a retailers site.

Heck, many legit top lists include dedicated YouTube videos visually inspecting the products and reviewing them - cross referencing blogs to YouTube channels and validating that the YouTube video is of some production quality (likes/views/youtube Spam detection) could strip down a ton of spam.

> Simply count the number of ads in the text.

What is an ad?

Something served from double click or other major ad network

That does absolutely nothing to address the affiliate problem, nor does it catch advertorial or "native ads" as they sometimes called. Ignoring all content served by ad brokers is no more sufficient than browser ad blocking extensions. It helps, but you don't get an ad- and distraction-free result.

Plus, as soon as you make that a metric that is penalised by Google, how long do you think it'll take until ads are distributed from some rotating nondescript domains?

(See Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure."

https://en.wikipedia.org/wiki/Goodhart's_law )

Don't spammers already do a lot of domain cycling? Doesn't NextDNS already ignore domains less than 30 days old, for exactly this reason?

Most affiliate spam sites that I see cover the page in intrusive advertisements to maximize revenue. Penalizing intrusive ads would certainly have some effect. This is also referring to penalizing rankings so that the entire page gets downranked rather than simply blocking ads in your browser.

It still wouldn't do much because as soon as Google starts to try to identify sites full of ads by some criteria, shady sites are going to start serving ads in some other way. It's already hard enough to distinguish ads in some sites that have gone with native ads or advertorials.

Reply on news.ycombinator.com