This isn't a conspiracy. It's basic business and doing otherwise would be the basis for a tort.
In Google's case, their officers are responsible for optimizing the mix of objective search results with revenue producing search results. That optimum can be described as just good enough not to drive too many queries away while maximizing clicks to their customers. There's no legal requirement or demand from shareholders for a wall.
And indeed the very idea of tailoring search results to an individual's past browsing history is always going to push sites that share data with Google to the top of the results page.
TANSTAAFL.
The constantly repeated 'duty to maximize shareholder value' line is nothing more than a myth.
In the case of Google, the triangle of Page / Brin / Schmidt basically control Google outright, regardless of the other shareholders, due to their voting shares. So they absolutely do not have any duty what-so-ever to maximize anything. Buyer beware, is basically what they stapled to the prospectus.
http://investor.google.com/corporate/2004/ipo-founders-lette...
In fact it's no more complicated than this: if some random shareholder is upset, they can stir the pot accordingly, and the waves they'll make is almost always in proportion to the shares they can vote directly or indirectly. There is no singular objective qualification on what would lead to the maximization of shareholder value, it's an opinion that varies from one shareholder to the next as to what they think is "best" for the company.
Simple example: some shareholders might think it'd be better to slash salaries at Costco to boost the bottom line. Others believe part of the reason Costco is so successful is their employee culture.
Short-term-ism in terms of maximizing ads revenue which costs long-term goodwill is a serious negative for Google.
I'd argue that in the past year or three, the company has started showing its vulnerabilities. I'm not sure who will take over from it, or how, or what that company's business model will be, but I see vulnerabilities.
The dynamic Google results page means that it's really difficult to refine a search based on the presently visible results which disappear as I update the search. I find that that behavior incredibly annoying, and greatly appreciate that DDG doesn't do this.
Or when I put an entire error message in quotes and it deems the number of results for that error message too low to be intentional so it deconstructs it into a useless mashy search of all the words in the error. Note, I don't mean when the results are zero, but even then I usually have to spend an annoying amount of time before I realize that my search actually had zero results instead of the millions it claims it had.
There was a time when google's cleverness was just enough to be useful, but it gets more and more clever (and frustrating) every year now.
It's much better than the first time around: more relevant, faster, and very few technical hangups.
I still fall back to Google periodically, especially for:
• Date-bounded search. DDG doesn't support this.
• Specialty searches: news, books, scholar. I've also keyed up custom searches for a bunch of sites in my browser.
• Rarely: I don't seem to find what I'm looking for on DDG. Usually first an !sp re-search, if that fails, !g. About 2 times out of 3, I still don't find what I'm looking for and return to DDG for more refinement.
Google has a dual class stock. The only shareholders with any power are Larry and Sergy. That was done to avoid short term thinking (precisely like you are proposing). Investors know this when they buy on.
This is a case where I posted in a rush. The basic idea that Google's ranking algorithm is optimized to serve Google's interests first and those of its customers second is the only possible way for Google's officers to fulfill their legal obligations to the shareholders they serve.
The key to understanding this idea of the best search ranking algorithm is that people who query Google's search engine are not Google's paying customers. Google Search's paying customers are almost exclusively advertisers.
The best search results Google can produce are those which maximize their revenue. Not enough traffic directed to ad buying customers and advertising dollars may go somewhere else. Sure too much obvious selling might drive queries elsewhere but the threshold for tolerating advertising keeps going up. So many people take tracking across sites for granted that Google can push a "weather" search onto an advertising affiliate's site and still meet the expectations of the data point making the query. There is no objective reason other than income for ranking secondary sources above the primary source, "weather.gov".
Occam's razor just cutts that way.
It's true that searchers don't pay Google money, and advertisers do. But Google is running a platform. In the past I've compared it to an information marketplace. And the goal for Google is to make the market run as efficiently as possible, otherwise they risk losing one side.
Searchers don't pay Google, but they do (presumably) pay Google's advertisers, who pay Google. If you lose the searchers, you lose the advertisers.
Now of course there is a balancing act, which you allude to in your last paragraph. But there are plenty of easy examples where Google returns no ads even though they could. A search for "how old is barack obama" just returns the number (or Wikipedia), without ads, even though I'm sure there are advertisers out there who would pay for an ad to be shown.
So obviously it's not universally true that "the best search results Google can produce are those which maximize their revenue." Perhaps adding some subtlety to your argument would help me understand exactly what you're saying.
For example Google Search might be considered a marketplace, but such an abstraction might lead a person to lump buyers and sellers into amorphous blobs and ignore the heterogeneity within each group. Ford and overstock.com are different sorts of advertisers and thus Google's business comes down to segmenting end users.
Plain and simple the most valuable end user segments are people who not just tolerate tracking and targeted advertising but who actually derive value from it. They are valuable not only because they click through and buy stuff but because they validate Google's claims that its business of tracking users and pushing ads and tailoring search results toward commercial interests and away from the long tail is objective.
Long tail results are not revenue generating and Google has simply removed bit by bit the end user's ability to specify them. Sure spelling correction is useful, until Google search refuses to respect quotation marks and simply renders some terms unreachable. Local search is useful, until a person wants to search across borders or outside their local language.