Google Books and the open web
sappingattention.blogspot.com
sappingattention.blogspot.com
I appreciate that Google and Bing spend millions of dollars trying to purge their indexes of spam, but they also seem to spend millions of dollars creating adaptive results based on an unknown list of parameters and variables that are opaque even to the search organisations themselves.
It's time a new search player stepped up with a proper mature indexing concept that allows for deep hierarchical filter searching based on a semi-static system that only removes results from search when they actually go offline, and doesn't prioritise social media posts above everything else.
We need to bring the internet back to the primary function of an information resource first and foremost. The clickbait-ad-selling junkie that it has become is creating a closed circle of idiotic users consuming and creating yet more idiotic content, like a fish eating its own tail. What's more, Google and Bing are the primary enablers of this whole problem.
I don't know how to fix it, but somebody has to do something before the internet becomes nothing more than an ad-ridden gogglebox in the same way television has already become.
That you presume they have any other goal in mind is frankly bizarre to me.
Through experimentation they have arrived at these algorithms to maximize the probability that you find what YOU were looking for. Based on who you are, what you've searched in the past, what context they think you're in, what's happening recently, what's happening nearby, etc.
If you want the search engines to ignore the context of the individual, then most people will get what they are looking for far, far less often than they currently do.
That's what Google and Bing are experts at optimizing.
There are search engines that don't try to optimize based on your personalized context. You and everyone else are free to use them and you don't because they're not as good at what you want. Statistically speaking. I admit there are edge cases.
Depending on the query, 60-80% of Google Image results point to Pinterest, which no one is looking for and is not the original source of anything that shows up.
For regular searches, in the past few years there has been a continuous and significant decline in both the accuracy of results and the ability to communicate to Google what you're looking for in the first place.
Quoted phrases are frequently reinterpreted, destroying the intent.
On a query with 3 words, Google will frequently ignore one of them entirely, based on an opaque and constantly changing set of rules, and will return results for the other 2 words that have no relevance whatsoever.
Perhaps most frustrating is the constant irritating battle to trick Google's many different query "filters", not all of which are NSFW "safe search" filters, to avoid the very strange situations where a query with a small number of common english words somehow returned no results whatsoever, despite Google's clear willingness to throw away even the most important words in a query in order to return something.
> If there were a better way to find the results that users are actually looking for, then Bing and Google would be working their assess off to find it.
I don't think we should assume this. I'd only be wiling to assume that Google/Microsoft would change how they do search if it could make them more money. That doesn't mean 'better' search necessarily. If they could convince users to accept it, I could imagine an extremely ad bloated search service, or search service that just sells the top ranks of terms to the highest bidder.
Sure, that's not 100% true, but I think people overstate the degree that that's not true.
Google and Bing can't convince users to accept something like that. In fact, people abandoned search services like that in favor of Google and Bing, because they don't do that.
Google is an advertising company. They are not a search company. There is absolutely no reason to think that their goal is 'good' results. Their goal is 'engaging' results.
If users believed they got bad results at Google, then Google could no longer show the advertising.
"Wishes of the searched" has several dimensions, and now goes far beyond any setting in robots.txt .
Imagine if you were looking for books about Lobsters, and you start finding books about Cows. All the books before were about Lobsters, so you assume you've ventured into the Cow section, and turn back.
If you are on page 12 and the relevancy of the results around you is low, you are invisible. Everyone has turned back by now.
Essential issues with comparing google's algo to any other ranking system is that the data they have on those pages is bad, so results are bad, so it's not as good as even a human manually sorting things. Obviously the ability to serve every query imaginable instead of blunt categories is hugely different, but I digress.
This is how, for example the query "that movie where there are two magicians" returns varied results about The Prestige, whereas "that movie where there are four magicians" returns various results about Now You See Me.
A pure text search would likely return very similar results, but the entity mapping heavily impacts the search results.