Study Suggests Google Harms Consumers by Skewing Search Results
on.wsj.com
on.wsj.com
No comments there, maybe some will arise here.
Either way the link to another similar thread is an odd thing to be tired of.
"The authors conceded that there were differences between their click surveys and actual search results, and that they didn’t have access to internal Google data to verify their results."
In other words, their conclusions are meaningless because they simply don't have access to to the data to determine how Google would rank search results in the fictional scenario that they were attempting to study. They essentially did A/b testing, showing some users fictional, Yelp-heavy search results injected into a fake Google results page, and compared those CTR's to those of an actual Google results page. While the fake pages achieved a higher CTR, that would probably be the case for many custom-crafted results pages curated by humans.
This was a "study" that was paid for by Yelp, in their quest to force Google to feature their results more prominently. I guess when you build your entire business on quasi-extrotion, heavy handed tactics come naturally. But this one falls completely flat for me.
It turns out what they did was use a site called UsabilityHub (which is intended for testing website designs) to show the study participants a screenshot of search results. One screenshot showed Google's results, while another shows search results that were purportedly sorted purely by relevance. I'm assuming users simply reported whether they would have clicked on any results on the page, but the article doesn't go into that level of detail. I also assume the users weren't told whether they were looking at the genuine results or not.
Given that, there's nothing at all surprising about that statement you highlighted. In fact, unless Google were willing to actually run the study themselves, randomly showing either their current results or results sorted purely by "relevance", and gathering the real clickthrough rates, then what the article describes here seems like the only real reasonable way to do this.
Therein lies the problem. They don't know how Google would order the results in the hypothetical scenario where results didn't include those that make Google money. This goes beyond not having clickthrough data; what they were showing to their test users was simply a not-very-good, human interpretation of what they presume the Google algorithm might return in a nonexistent situation. Unsurprisingly, these fictional search results had worse CTR's than were achieved by displaying presumably custom-crafted Yelp results.
The term relevance is unique to each search algorithm, and these researchers did not have the data to determine what the relevance would otherwise be for Google. This report essentially says "if we ran Google, we could get better CTR's by showing Yelp results". That does not prove consumer harm.
They kind of do. Google's very public about most of how they sort results, and the article said they used Google's "algorithm" to rank them. I know Google used to even expose the underlying PageRank number for various results, but I don't know if they still do so today (if so, then it becomes even easier to sort them the way you think Google would).
> what they were showing to their test users was simply a not-very-good, human interpretation of what they presume the Google algorithm might return in a nonexistent situation
You're making a very bad assumption here. From the article:
> One set of users saw a page reflecting results currently displayed by Google
Based on that it sure sounds like they actually performed various searches on Google, screenshotted the results, then fabricated a second set of results that reflects what they believed Google would have shown if it sorted purely by "relevance" instead of prioritizing its own services. And these fabricated results had significantly higher CTRs than the genuine results from Google.
> The term relevance is unique to each search algorithm, and these researchers did not have the data to determine what the relevance would otherwise be for Google.
Sure they do. The way Google ranks search results is quite public and has been so from the beginning. What isn't public is all the various tweaks they perform to the results, such as removing what they believe to be spam or content farms, as well as the biasing they do in favor of their own services. But the core algorithm is public.
> This report essentially says "if we ran Google, we could get better CTR's by showing Yelp results". That does not prove consumer harm.
The report says that they believed that Google was suppressing results from "third-party review sites such as Yelp and TripAdvisor" (not just Yelp) that it would have otherwise deemed to be relevant, in order to prioritize Google's own services, and that this artificial adjustment of the search results is harming consumers because it's yielding results that are less useful to consumers (as measured by CTR). Which is a bit different than what you're saying.
And this actually makes sense to me. When I search on DuckDuckGo for places in my city, I often get yelp results near or at the top. So they're obviously deemed relevant by the unbiased ranking that DDG uses. And I often get Yelp results if I search on Google as well. So obviously Google thinks Yelp is relevant for many searches. So if I perform a search that Yelp has relevant results for, and Google isn't showing me those results but instead shows me e.g. a Google+ business listing, then it seems quite reasonable to claim that Google is artificially suppressing the Yelp results in that case, and to further claim that this is causing consumer harm because the Yelp results were more relevant than what Google chose to show instead.
Yelp obviously wants to monetize their (ummm, sorry, users content) content. I would think company like Google wouldn't have much problem paying Yelp but probably their demands are too much (just my guess). So Google showed them the door and decided to do their own thing and now Yelp has problem with that.
For companies like Yelp, I would suggest they respect users intent and rights. Millions of users have sweated on to create all these content on Yelp. They shouldn't be building a giant walled garden around it. Instead they should just make all data available with a fair license to anyone who wants to use it. When more companies like Apple, Samsung etc buys their licences, it might be overall better for them than to get one huge deal out of one company. May be they should even make their data available for free for research (they did made small part available, but I'm saying why not all of it?). When the community works on this data, lot of cool ideas can spark which again benefit them to get more licensing deals and would make it worth for users who sweated out to create all this content.
The way the US military works in large and increasing part is that it works with US corporations to achieve these effects. Given the complaints by other countries of weaponization of Google/Facebook and studies done on using search engine results to manipulate voting we should keep our eyes peeled that Google continues to 'not be evil'.
https://www.google.com/search?q=61+million+experiment+social...
There are also - and I can go on about this for quite some time - programs and studies partially revealed by the Snowden docs but evidenced well beyond those documents that have the US investing resources into not only tracking the flow of ideas using large scale sentiment and social media analysis, but also curating and directing these ideas. While there is around 10 years of capability research regarding the online case, today DARPA has a research program called SMISC (Social Media in Strategic Communication; Strategic Communication is the DoD term for influence operations/propaganda) that studies how to measure the 'idea posture' of an online community and also how to control it. This combined with efforts like Operation EARNEST VOICE and the host of others does make an onlooker take the State Departments 'War of Ideas' concept much more seriously.
On what basis do they decide whether results are harmful or helpful?
> The study’s authors—[…]—found that users were 45% more likely to click on results that were ranked purely by relevance, rather than as Google ranks them now, with its own services displayed prominently.
If users are clicking on results 45% less often, that suggests they aren't getting results that are as useful to them. From that it seems fairly reasonable to claim that this is harmful to consumers.
* Google is a publicly-owned corporation. They have every right, as they own the platform, to skew it as they see fit. They aren't supposed to be unbiased, if you wanted that you'd have to have a completely open-source search engine to make this argument credible.
* Just because they own the lion's share of the search market does not mean you have to use them. Don't like Google? Use DuckDuckGo. Use Bing. Use AskJeeves! Are people using Google at gunpoint? Certainly not.
This whole things feels weak - isn't it obvious that Google is owned by a corporation with it's own self-interest agenda, and by using that service, you are bound to the skew it may or may not put in it's searches / ads / everything?
It couldn't possibly be that Google did anything wrong.