What's frustrating is that these often seem to be a handful of domains, like 'geeksforgeeks' and 'towardsdatascience'; for a while, of course, there was also the "gitmemory" spammer who seemed to be able to push out Github results on both DDG and Google. Yet I think Google removed reporting and blacklisting domains from searches long ago, and I think DDG never had it, leaving the only option for removing them client-side scripts and extensions that work poorly. Likewise, no search engine appears to be manually blacklisting them. Yet as you point out, if one did, then the spammers would probably just move to using many domains, which would probably be worse.
A fresh domain that suddenly has a ton of content should be viewed with suspicion and downranked as it's likely a spammer (copying GitHub/Stackoverflow/official docs).
Legitimate sites that are starting out shouldn't be affected as they are unlikely to have a ton of content from the start.
Of course, this isn't perfect, but it should take care of the majority of spam copycat sites.
And what makes you think search engines don't take quantity and breadth of content ? And, why do you think SEO won't be able to take advantage of that ?
Most people commenting and lamenting about search quality need to understand how much of search quality is an adversarial game, that neither side (search engine vs SEO people who want to manipulate the result in their favor) can win decisively forever. And generally it's not for the lack of trying the quality hasn't improved as much - it's just some areas it's more difficult to make progress due to more money spent on making it difficult for the search engines.
What I am most disappointed about Google and search engines in general isn't so much about ranking simplistic things like generic recipe or product reviews and stuff like that - most recipes for general items are all similar, and product review is inherently commercial that reviews always have been biases (short of using Consumer Reports, almost all magazine review articles are biases and influenced by manufacturers). What I am most disappointed is the lack of improvements in UI and retrieval/recall. Interactive UI that can narrow down search space iteratively and interactively so that I can describe what I want in more precise way is lacking. Where is the virtual librarian that I can talk to to narrow down what I want in a step by step way ? Instead of a single search query being the only input, why shouldn't search engine ask for clarifications and let users more precisely describe what they want ? Human languages and human brain just don't produce a single phrase to describe what we want - it takes sentences and questioning back and forth to determine what we want, and we often need more input to realize and describe what we actually want. And all search engines have failed to provide such an interface. And my bet is that whoever can crack that would beat all other search engines, since users can give better, higher quality input for the search to find.
I remember using an open-source ML model (trained on Buzzfeed article titles) to detect YouTube clickbait based on titles and it worked brilliantly, and that was just downloading some code on GitHub and running it as-is. I'm sure the same could be applied to search results and you could achieve much better quality if you actually put some effort into it.
I very much doubt this is some kind of hard problem as opposed to Google just giving up because their business model doesn't actually incentivize good search results.
Link to the Repo please?
Google could start incentivizing high quality unique content and websites from domain experts, but they have decided they can’t make as much money off of independent publishers as they can from marketing/content farms.
An algorithm could punish things inherent to SEO optimized sites (ads and tracking). Removing the ability to passively generate money without providing useful content is the key.
Of the top of my head, a system like the following would be a pretty good start:
Scale of 0 to 100 (closer to 100 being higher up on the results).
You are penalized for the following:
- subtract 10 points for any ad, using ublock origin's list as a good starting point (this stacks, it's 10 points off for each link)
- subtract 100 points for google tag manager
- subtract 100 points for the facebook like button
- ... etc for each of the major tracking scripts
This would obviously need to be updated as ad-tech evolves, but it would cut out 90% of the current SEO spam.
Can google do this? No, they have a conflict of interest around placing ads. Somebody else, however, can absolutely do this.
The Google competitor will have a human nuke their entire domain or business (based on a manual index of banned products/brands) from the search results forever, or have by default a bias against ad-filled websites which would remove any commercial incentive for those websites to exist in the first place.
But yes, supporting community-supplied adblock-style lists would be a start, and Google isn't even doing that.
Maybe it's an idea whose time has come.
I personally just append `wiki` or `reddit` to queries. Crappy but kinda works.
The problem with Google is that their business model is to show you ads (either on their own website or third-party websites embedding Google ads/analytics) and not to provide you quality search results, therefore they have no incentive to combat even the most obvious SEO spam (see Pinterest & image search for an example).
It won't be perfect, but at least Pinterest wouldn't be polluting image search for years for example.
The list of affiliate marketing sites might need to be human curated (just like SponsorBlock) to avoid false positives.
They all use different algorithms, so the SEO specialists can't game them all.
Making crappy sites would maybe no longer be worth it.
Real competition in search is probably the best way of reigning in the SEO sector.
Google would need to not have a monopoly if they wanted to do that. Otherwise, they would be accused of anti-competitive practices (since their policies would be aligned with their policies in other services, they would rarely ban themselves).
If there were multiple search engines in use with different ranking algorithms and different search strategies, then "SEO" would be difficult to impossible to achieve because optimising would one search engine would likely be a pessimisation for the others.
We don't just need one competitor, we need many. And they can't just be copycats, they need to do their own thing, and that might well make the cost:benefit of SEO unviable.
This was met with a second instruction, the first five people in line, move to the back of the line, and last five in line move to the front.
it became a trial and error, with the usual kids jostling for position now wanting to be, at the back of the line, then in the middle of the line, hunting for the condition that creates pole position.
the idea that it was the personalities and the value sets, not the position on the line, that triggered the condition, seemingly was too abstract to be deduced. keep in mind this was grade 2.
so may the browser filter the first to the nth SEO spammers in the query results be sent somewhere away from the front of the line until a match for query terms occurs.
To me it seems to be compelling evidence that search engines are not a viable way to organize information on the web, but that's a different topic.