Hacker News Reader

Comment by Beltalowda | Hacker News Reader

> So, that’s another opportunity for DuckDuckGo, by ranking the results based on the content’s quality and relevance, as Google used to do before.

This makes it sound like that's easy. Yes, SEO twattery is an issue, and with the growth of the web it's also much more prevalent and the stakes are higher. Google from 2022 has a much harder problem to solve than Google from 2000. DDG can't "just" solve that just as Google can't "just" solve that.

View on HN

I disagree; Google isn't even trying - after all, Pinterest is still polluting image searches despite not even trying to do anything malicious.

However even if we assume that detecting spam content is hard, detecting the funding of said content is much easier - most of this content is there to get you to buy something or look at ads, and they use analytics tools to track conversion.

Use the presence of ads, analytics, affiliate links and other marketing tools as a proxy for "spam" and downrank them, or offer the user a "non-commercial" option like Kagi does.

Now you move the problem from old-school SEO "make my content rank higher" to "conceal my ads/analytics/etc from the search crawler so I don't get downranked". That is a very effective strategy because advertising has to be served via third-party code to protect against ad-fraud, but third-party scripts are trivially detectable.

Now the spammers and their partners have a problem - the only way to fool the crawler is to self-host the ads, but doing that would expose the advertisers to ad fraud. You're pitting the spammers against each other in a game where the only winning move is not to play and the extra overhead will make the spam less profitable (or not profitable at all).

99.9% of the web uses ads, analytics, affiliate links. I don't think anybody is interested in a search engine that only looks at 0.1% of the web.

This is the case because we optimized for it with a profit-centric internet.

If search engines start penalizing this kind of content then the 0.1% might grow. Not to mention, most of my web search usage nowadays revolves around open-source library documentation, blogs, GitHub and occasionally StackOverflow, so being constrained to the 0.1% of the non-commercial web doesn't appear to be a downside at least when it comes to searching work-related issues as a programmer.

A lot of the high quality content that exists on the internet only exists because the advertising pays for the people who create it.

Be careful what you wish for. If you make sites with ads rank lower the void will just be filled with crypto scams or affiliate marketing. Remember it's a lot easier to make a spam site than a high quality one, changes that disincentive high quality sites from existing won't necessarily do the same for spam sites.

Quantity of tracking and advertising is a great proxy for spam content. If a page contains 10+ affiliate links, I don’t want to see it. If it contains just one it might be more legitimate content like an author talking about their book and link which gives them a commission on it.

Look forward to multiple sites with just one link on each page then. Any kind of well-intentioned rule you set can be figured out and gamed.

I've been using crowd-sourced tools like SponsorBlock and Return YouTube Dislike, and I really like them. So far they've resisted attack using some kind of consensus algorithm. I've also been using the search engine Kagi, which lets me downrank and even eliminate certain domains and sites which I believe to be low quality. I wonder if it's possible to marry these. If the majority of users dislike a domain, it gets downranked. Of course bot detection and perhaps the use of logins might be required, but compromises must be made if the desire is a better search engine.

Such shifts are’t free. If spam is less profitable, you get less spam.

I follow a lot of YouTube creators who product genuinely excellent and valuable content. They typically limit their sponsorships to one short segment per long video. At the same time, their video descriptions often contain numerous affiliate links, preceded by a disclosure that the following links are to be used if you’re interested in purchasing one of the products and you want to support the creator.

Those links honestly do not bother me at all. I barely even notice them since video descriptions are hidden on YouTube, apart from a few sentences at the top. So how do we make search engines that distinguish legitimate creators like that from spammers who create fake blogs with fake reviews and tons of affiliate links, all using a fancy blog template that makes the site seem trustworthy?

YouTube descriptions started being useless since they became a copy/pasted generic blurb of text (irrelevant to the video at hand) and a dumping ground for various social media or similar links, so no surprise that you aren't bothered by spam links in something that nobody looks at anymore because it's already been spammed to hell. But there was a time when descriptions were all manually written and were an actual summary/overview of what's in the video.

> So how do we make search engines that distinguish legitimate creators like that from spammers who create fake blogs with fake reviews and tons of affiliate links, all using a fancy blog template that makes the site seem trustworthy?

It's a whole other problem for videos so I'm going to approach this from a website search results point of view: we simply use presence of commercial content as a ranking factor - if the commercial content is all that matches your query then you get it, but if something non-commercial matches your query equally well then it comes first. This encourages non-commercial content without banning commercial content entirely. Ads/marketing/etc are noise, so it would be normal for a search engine designed to serve the user to rank low-noise content above noisy content, right?

I don't know what content you consume, but most commercial content I find is garbage, while non-commercial content shines because it's produced by someone who actually cares.

First, what's wrong with having Pinterest images in the search results? For the types of queries where they show up, they usually seem to be pretty relevant.

Second, many high-quality sites use ads and analytics. What are you going to do? Downrank the New York Times because it has ads and uprank your local conspiracy theorist because he doesn't?

Why would you want a pinterest reupload of an image, instead of the actual source?

They've played games in the past with requiring a Pinterest account. Same with the Quora spam.

Pinterest hides the actual image from the search results in a sea of unrelated images, and blocks you from accessing the images without a login.

It’s impossible to find the actual image from the search results if the source is pinterest now that Google removed the direct image link from the search results.

> Downrank the New York Times because it has ads and uprank your local conspiracy theorist because he doesn't?

If the only difference in ranking between the NYT and the local conspiracy theorist is the presence of ads then yes absolutely. Realistically speaking there's no way the local conspiracy theorist will get the SEO "juice" of the NYT (including domain age/reputation, backlinks, etc), but if he does then he deserves his place. I don't want a search engine to be the one deciding what is considered the "truth".

Yes. Why hadn't I thought about it? Everyone talks about how easy would be to solve this problem. Well if it so easy do it and become the next Page and Brin, instead of suggesting other to do.

It's humans (SEO) vs AI (Google). Humans win so far.

Let users just rate that search result, so if someone else searched for the same thing and didn't find it relevant, they can thumbs down. If they found it relevant, they can thumb up.

Would the votes apply only to the specific search query?

Site reputation score could be factored in.

There should be a relevance score, and it might be cross-referenced to other similar search queries that mean the same thing, but that's a machine learning project all on itself

That only works if people are using the same search term AND looking for exactly the same thing... If they're looking for different search things but using the same search terms (i.e. some people are using sub-optimal search terms), that's just going to add effective random noise to the ratings.

Given how location-aware some things sometimes need to be (including spellings / alternative words, which in my experience, Google is a lot better at than DDG), I don't see how that works at all.

Nobody said I can't use a SIMILAR search query to have a slightly lower bound for relevance. Machine learning can identify if two queries mean a similar thing in the context of search

What makes you think a user rating system wouldn't be gamed?

Especially since it has been tried and that is exactly what happened.

Exactly this. Pagerank is/was another form of voting.

You don't just average user ratings, you need to have a smart system.

First, each user needs to have a history or they won't be considered in the rating almost at all. Then, each user's history gets analyzed for average rating vs. the average userbase so you can determine whether this user upvotes everything or only rates the sites that are terrible.

So each user gets a score of "threshhold" of what rating site they will thumbs up. Some users will thumbs up a site that's 50% (mostly useless, but not outright spam), while some only upvote sites that are 80% (good content at the very least).

Then regressed for this variable, you can calculate their feelings on each site and see if it correlates well with other users. The more ratings, the more each additional rating is weighted.

This will reduce the number of ratings required for a statistically valid sample for each search query (since you need to tell users if it's relevant for the thing being searched, not just quality of the site). But you also need to fight bots as well.

This system may not be perfect, but it would add value to a search engine

Mate, I'm going to pay $1000 and get a bunch of Filipinos in a LAN centre using that SaaS company that routes your requests through other people's VPN software to click a bunch of thumbs up. Watch as my cialis video tops your search for EC2 instances.

We already have that. At least I can hope for a variety when random users are also involved.

My Filipinos will win.

Not OP, but allow users to follow others and create an aggregate filter. That leads to its own problems, but if results are done with a weighted preference depending on if have been noted as "good" by people I've followed, or to a lesser extent those they've followed, it may produce better results.

I’m not sure the solution is to create a search results social network…

Why? Perhaps only add people you know and trust IRL, or have some other sort of reputation system.

That’s way too much effort to use a search engine. I’m very technical, and I don’t see myself being interested at all - forget mainstream users…

You need to do it with user accounts so the search engine team can remove fake clicks and ban the user. Users with a low amount of ratings won't have any say in the rating of the site, since they will have a low confidence.

They will first need to rate a few hundred sites (valid ratings, because if they don't have high validity with the rest of the sites in the database the confidence this is a valid user will be low) to give the spam site a high rating. So they will have already built a database of good ratings before they can give the last vote that they get paid for?

Also, you can manually remove a spam site that uses this tactic and all their money will have been wasted.

Even better, let users leave comments on on search results. That stops all the "search query +reddit" searches because now Google is reddit. The discussions are happening on the search engine response page.

Reply on news.ycombinator.com