Dear Google, I am writing an open letter from the search wilderness | Hacker News Reader

Dear Google, I am writing an open letter from the search wilderness

tnooz.com

41 pointssunnykgupta84 comments

TL;DR: Lamentably, Google has facilitated a transition to organic search results being strictly a money game for most categories, removing the joy of discovery and disproportionally impacting small businesses.

What's missing is a suggestion of how to fix it. Not an easy problem.

I was wondering about this the other day, and considering another approach to search, which I'm sure others have thought of too as it's fairly obvious. To restate the problem:

For any given search term there are millions of hits

Of those hits, only the first 20 or so in a given ranking will ever be seen

This renders a huge amount of the internet invisible and a huge amount of indexing/linking moot

At present google uses what sounds like a very complex mix of signals about linking, reputation, novelty etc which it constantly tweaks, and delivers a relatively stable ranking which guarantees that those with deep pockets will win the game of ranking highest simply by buying top position or buying authority in various ways (sponsored blogs, buying authoritative domains, content farms etc).

Another approach would be to constantly inject random search results for a given term into results, and use solely reader reactions to results to rank them (registered user voting, do they come back to search results or not, sites chosen as high quality by this user, that sort of signal). So always have some randomly chosen results to guarantee mixing, and always include some degree of user input to give better and more relevant results chosen by humans. I think you could build good results like this, while completely ignoring any sort of complex ranking algorithm which is both too stable, too prone to gaming, and in conflict with the revenue stream of ads. Obviously you'd have to be careful about which users you listened to, but that is a solvable problem if you can persuade people to register to get better results.

It's definitely well past time for a few serious non-corporate competitors to google to emerge, if only to keep Google honest.

how would you track the user inputs in order to create then rank the results further? via browsers? or...?

while I like your idea, I'm not sure if it is feasible and second, if there is an 'acceptable' method which most people will not object to or try to game.

for example, imagine 1000's of mechanical turks being paid cents to click and browse to a specific link in order to fool the algorithm you've set up

Tracking users and combatting fraud would definitely be an issue (just as it is for google), and you might have to set up trusted rings of users or some sort of web of trust rather than just trusting any user. It'd be interesting to try - tracking would have to be by browser cookie and actions like returning to results or upvoting/downvoting on the results page and you'd have to be very strict about fraud prevention from the start.

Perhaps more interesting than a specific ranking algorithm though is the idea of injecting random results into the mix. Perhaps Google has tried all these things already, it wouldn't surprise me and I know they look at user reactions like returning to results/time on page etc, but it'd be nice to see some search engine which is not owned and controlled by a massive corporation like MS, Google or Apple though - is anyone doing something interesting with common crawl?

There's a German metasearch engine funded by an NGO & University, which also funds the Yacy P2P engine: https://www.metager.de/en/ and http://yacy.net

Scholar Ninja is trying to use WebRTC for distributed indexing and searching from a Chrome extension, but there's a Chrome/WebRTC bug which prevents long-running tasks: http://juretriglav.si/an-open-distributed-search-engine-for-...

Thanks

The problem with injecting search results is that most of the time, it is going to give users a worse experience, not a better one.

I'm not convinced that it would - is there a qualitative difference between page 2-3 or even pages 1-10 of Google results and the first results on page 1? A random mix of the first 10 pages of results say I'd say would be more interesting than their current mostly static and gamed results, which are basically pay to play.

The most valuable thing about this idea is that it is immune to gaming because of the mixing. Just as an example relevant to the point of the original article, here is a search for hotels in Devon on google:

https://www.google.co.uk/search?hl=en&q=hotels%20in%20devon

First half of the page are links to Google places entries - so advertising for Google. Are these paid for? I'm not sure what the situation is there but this is partly a manual intervention from Google to try to save their results... After that on page 1 we have lots of aggregator/review sites like laterooms, tripadvisor, stayindevon etc (your results may vary). by the second page of results we have some real hotels, by the third, we have more real hotels, etc, up to page 9, which is still real hotels in devon.

So I'd say the first 10 pages are pretty much equally useful, and some mixing of results would do wonders for ranking and allow good results to surface instead of languishing unless they make an entry on a google property, or on one of the portals.

I like your idea. A small amount of randomisation can be very effective at (1) defeating gaming, (2) finding unexpected signals in complex systems (3) Preventing over-fitting of results when optimising.

All three of these advantages come into play here.

Good analysis

Without going into too much details, just randomly mixing the first ten result in the first page will totally destroy Google's results: anyone will notice in a matter of days, if not hours. And everyone will find Bing so much objectively better that it wouldn't even be funny.

For example,

"rosetta spacecraft": the first two results are from ESA's homepage, the tenth is an article from The Guardian.

"mauna kea weather": after a gigantic weather box, the first result is from Mauna Kea Weather Center, the tenth is a page from some generic website that deals with all mountains.

And on and on.

Without going into too much details, just randomly mixing the first ten result in the first page will totally destroy Google's results

I didn't suggest doing that.

Which results users click on, and especially which results they click on and then don't come back is a huge input into Google's ranking algorithm.

In fact, this is pretty big barrier to entry when creating a competing search engine. Google's scale enables them to get a much better signal from this input than anyone else can.

Yes I've read about that elsewhere, but they do still use a lot of other signals AFAIK. The more complex their algorithm becomes, the less inclined they will be to change it, and the beauty of only using user input is that you get an ever changing signal and people start to be less obsessive about their exact ranking, because it changes constantly. Of course it does introduce problems of its own with trust.

Re competitors, I think that's more about eyeballs than quality - Google is such a behemoth now and is entrenched with revenue from advertising - they can use their other properties and interests like mobile to feed search revenue. I really don't think they're at the top solely because they return the best results any more, because they've been without significant competition for so long.

I was actually more interested in the idea of injecting some random results (say from the top 10 pages), as that might significantly increase the relevance of their results for many queries, give them even better data from user input, and would make it very hard to game the system.

The idea that you could only use user input is not realistic. 15% of Google's incoming queries have never been seen before by the search engine. There's not enough user input available.

Not a solution, but a thought: it's easier to game a single search engine that owns the market than it would be to game 3 or 4 competing and different products with a significant market share each.

Google's success has its downsides.

TL;DR: Google is giving users what they want instead of what I want, please change this.

Seriously, what this guy wants is a search engine that promotes what he wants over what users want. He fails to understand that Google became popular specifically because it didn't do that.

Sensible summarisation.

He goes a little further: he's objecting to the fact that Google is not acting as a "leveling" force to break up incumbent industries and change society. He appears to want them to act as a far-left government.

I don't really feel a need to rebut this desire, beyond saying "no".

It's kind of fascinating how the supposed economic and social benefits of the open web that people spent 20 years or more evangelising are dismissed as 'far-left' once powerful incumbents have cornered the market. Suddenly advocating for the destructive forces of open competition is the same as being a radical socialist.

The problem here isn't that there isn't open competition. The problem is that he's lost at it.

The problem is that it isn't competition on services or products, it's competition in gaming and/or just paying massive incumbent middlemen.

Who provide value to the consumer that individual business sites do not. Why shouldn't they be more highly ranked?

>Who provide value to the consumer that individual business sites do not.

You're reciting this as if it's obviously true, yet offering no evidence. If a scraper beats a site, it's not a logical conclusion that the scraper was more useful to the consumer than the original site.

How many hotel sites offer price comparisons to other hotels nearby? That's value right there.

This is succinct. It appears that the Google we had 10 years ago was perhaps valid in light of our ability to capitalise on 'the most valid results' and the Google we have today hasn't innovated on their search platform to break away from 'what corporates want us to find' into 'what we want to find'. We should expect more from our worlds most used search provider.

The travesty isn't even perhaps brought about by Google directly. It may in fact be a reluctance of people to question the status quo or at least be heard in doing so. If nothing else, this blog is exactly what people need to deliberate over without prejudice toward the writer, and perhaps some knowledge or value will come from that.

Whatever it is, we're left with a fairly opaque 10 site list that people rarely delve beyond. And I hear the complaints already from some, sure it ever so slightly customisable, but I certainly haven't cared to do so for years.

Recall that Google's initial and ongoing innovation was moving away from what corporate entities want users to find. I think Google prioritizes what they believe users want to find and subject it to the review of user behavior.

The perceived problem with this is that what they think users wants doesn't always agree with what someone else thinks users want. Like in this article, where many users clearly want to see aggregators but the author believes they don't want to see aggregators.

I think you will find that the reason why big brands are big brands is because a lot of people do, in reality, want to find them. Just because I don't want to find the same things doesn't stop that from being the most popular choice. Google would appear to be delivering what the bulk of its users want to see.

He's raising the observation that the easier it is to game Google, the less useful Google becomes.

He's not prescribing a solution. Much less the solution of Google becoming a "far-left government."

But screw equality and open access, right?

Search engines are by their very nature un-equal. There's no way around this without removing the value in having a search engine.

Some search engines, e.g. Google 7 years ago, are less un-equal than others, e.g. Google today.

The job of a search engine is to second-guess your desire from limited inputs and try to fulfill it. I can't think of any sane way to be "equal" about that without relying on a lot of subjective judgments. Or being bad at making decisions.

Is the strategic priority of Google leadership a subjective judgement? Are these subjective judgements objectively implemented? There are always winners and losers for any given ranking algorithm, the question is whether perception of neutrality has improved or declined in the last 7 years.

What is the perception of Google's neutrality in enforcement of rules against traffic arbitrage, e.g. search toolbars: http://www.benedelman.org/news/012213-1.html ?

Almost anyone who loses out will perceive things as being subjectively against them. Almost anyone who wins will perceive it as being a sign of their worthiness and objective value.

Yeah, but who is making those decisions?

Google? SEO agents? Large multinational corporations?

Wikipedia is subjective too, that doesn't mean that they just give in and let PR people manage the thing.

Among other things, they pay attention to whether or not the user comes back to look at results again. So in a significant way, it's the market.

His point, and something I agree with, is that the easier gamification becomes, the less useful a search engine becomes.

I don't know if this is true for everyone. It probably isn't. (at least it isn't immediately obvious.)

But for the author it is, and I agree wholeheartedly.

> He's not proscribing a solution.

The word you want is "prescribe." To "proscribe" means to make forbidden or illegal. https://www.google.com/webhp?sourceid=chrome-instant&ion=1&e...

Whoops, thanks.

Read the article carefully and you'll see that he does in fact prescribe a "solution": take search results away from entities who have lots of them and hand them out equally to everybody, so that everybody gets a "fair" amount of users directed to their site.

...which would of course render it utterly useless to those users and destroy any value it has as a search engine.

He even states that his main objection to Google search is that it is not acting as an artificial "leveller" to change society.

This is the manifesto of a far-left government. Google doesn't do these things because it's neither far-left nor a government, and his request is both poorly thought-out and quite bizarre.

I don't think that's an accurate TL;DR.

He's saying that Google is a plutocracy, not a democracy.

I have an joke/axiom for social networks: "It's not really a social network if you can buy your way onto it."

A search engine follows the same principles. If I can pay a SEO professional to have my results promoted, then information is not truly being organized.

Your logic is flawed. Random results (which can't be gamed) are more organized by your standards.

I don't think so. It's the fact that results are organized what makes SEO possible.

This is a good point.

I don't think this statement is necessarily a good and logically sound axiom, and I think it applies to social networks more than search engines. I view it as a sort of a zen koan, in the vein of "It can't be all that avant garde if you're using a french word to describe it."

At the same time, I think this bears more scrutiny. Intent is important here. Your assumption is that Google organizes information, and that companies are organizing their information in a similar vein in order to facilitate better indexing. If this perfect world is the case, than your argument holds.

Unfortunately, I think the reality is frequently different. I think the vast majority of SEO use cases are not the organization of information, but rather the promotion of profitable information and damage control of potentially harmful informaion.

You may say to this that analyzing the motives of SEO is a deeply flawed venture, and you'd probably be right. But it is the difference between a system for organizing information and a beaurocracy which only a select few have the resources to navigate.

Actually I completely agreed with the social network idea, I just don't think it translates well to search engines. It would translate to evil-IAP games ;)

SEO is just gaming the system based on how search engines organize their corpus. It's a catch-22 situation, if you organize information, then somebody else can discover patterns in how you organize it and game the system. Based on that, I'd say SEO is evidence of organization.

To be fair, Google results have been very accurate for me lately (probably knows more about me than myself), specially since they use click and return rates in their ranking algorithm.

Is SEO really such a game-changer for non-tech people? Or am I being victim of SEO tactics without even being aware of it?

What amazes me is that you can actually "buy your way into Google" by paying ads... and it's the base of their business!

Bear in mind that SEO is the sort of thing that gets used as a bugbear to explain why a given business is not as popular as the owner feels it should be. Sometimes the problem really is SEO. Often it's something else.

By way of comparison, I read a story not long ago where a restaurant owner blamed some incorrect hours of operation data in Google Places for a major drop-off in his business. Yelp reviews, on the other hand, told a story of food and service quality going downhill...

Actually, what I read him saying is that none of the businesses in his industry can get a listing within their industry. His point is that intermediate "marketplaces" inject themselves into an industry and then control consumer access to that industry's products. These "marketplaces" are not actually in these industries - they are in the internet business aggregate industry, and serve the consumers very poorly, because after all the consumer is NOT their customer, their customer is their investors.

He's not happy that aggregators are outranking the hotel's official website. I think this is a very valid complaint. The money aggregators take is used to fund Google visibility (paid or free) rather than for the hotels to use on other things.

There have been times in the past were Google's listings definitely were bad, like the whole Demand Media/content "farm" fiasco. The model of writing low quality and outright false content has largely moved from the how to articles on to news now, for better or worse.

The bigger issue is mobile means less results. A few ads and one or two visible organic listings. Google has to milk the hell out of desktop users to make up for lost revenue. If we move to voice search its going to really be different. You either dominate or get nothing, no in between.

Google has a model that works and makes a lot of revenue. They will not and can not innovate because those innovations will destroy the model. It is the same for any entrenched legacy corporation. Search is a good area to do innovative work in today.

Update: One other note. It was always clear to me that Google's business model was to provide huge volumes of free traffic, and then remove it. If you didn't buy traffic and you relied on that free organic stuff you were asking for trouble. Many companies have ranked organically for a term. Expanded, took on long term liabilities, and then lost their rankings and failed. Most of the business owner Google complaints we hear come from those type of people. You give someone a free drip of heroin, and then take it away and ask them to pay. They will not be happy.

Often, aggregators are more useful than any given hotel's website.

If I am looking to fly somewhere, I probably don't want to be taken to United's or American's homepage. I probably want Orbitz or Hipmunk or some other site that will give me the comparison tools to make my own decisions.

His mistake lies in assuming that a hotel's site offers more value to users than the aggregators.

You are quite correct on some points. If I book a cheap hotel for a few nights in a city even I use booking.com. If I book a weeks holiday for my family in a villa rental in far flung destination I want something else and I want to speak to the people with knowledge. Sad maybe but a lot of us like this!

The prevalence of aggregators suggests that a great many people prefer a system that lets them compare many options at once. That may be followed up by researching a handful of likely-seeming options more deeply, but few people with total ignorance are going to want to go directly to that. It's too time-consuming to start there.

Well perhaps there's a bit of that, but perhaps the value add of a well researched and more personable site fits far better, but people are too often expecting choice as the number one driving force in their decision making process instead of being asked to make appropriate decisions on their own wishes.

For those people I suggest they consider their fundamental reason for taking a vacation in the first place.

I think the prevalence of aggregation on holiday sites is simply causality of these companies not having a clear direction for their business. After all, it's easy to aggregate everything with the money and time to do so. But where is the value add?

Being able to compare a lot of options and filter down to just a handful that interest you is very valuable. It handles discovery and filtering steps in one go. There's your value-add.

You're asking for consumers to make significant decisions in what to them will feel like a position of ignorance. And you're asking for it because you think it will be profitable for you. Do you understand why I'm skeptical of this?

I don't think anyone is saying aggregators should not show up.

Do a search for a local hotel. I just did one. I counted 3 aggregator ads above organic results. 6 aggregator ads on the side, 2 other hotel ads on the side.

Organic listing #1 is the hotel, that is good. 7 other results are aggregators. 1 is the hotel parent company, 1 is an unrelated hotel.

What happens if you are a niche travel site, or a local blog? Your not going to rank. Your business model is now more or less dead.

I don't see the problem with that. Not all niche business models should be assumed to be indefinitely viable.

>Google is giving users what they want

In what way? How did the size of SEO and marketing budgets somehow become the result of consumer desire rather than a manipulation of it?

His complaint seems to be that users are shown links for aggregators rather than links for individual hotels.

I don't know about you, but when I google "Las Vegas hotels", I don't want to be shown Rio, Palms, Bally's, or Mandalay Bay at the top. I want an aggregator so I can compare and make my own decision. The same applies to flights - there's a reason I use Hipmunk and Orbitz instead of united.com.

Aggregators can offer more value to a shopping consumer than a hotel's site does. That's why Google is giving users what they generally want - a shopping user wants the ability to make comparisons.

>I don't know about you, but when I google "Las Vegas hotels", I don't want to be shown Rio, Palms, Bally's, or Mandalay Bay at the top. I want an aggregator so I can compare and make my own decision.

I don't. I'm not sure why you assume your opinion is universal. I want a bunch of hotels and their marketing pitches, then I want to go to my favorite aggregator to price compare.

All having the aggregators come up does is create an aggregation tax that every vendor has to pay - and each aggregator gives me a limited view of the market while giving the impression that it is the entire thing e.g. the lowest price on Orbitz is the lowest price amongst the people who pay Orbitz, not the lowest price, and since Orbitz takes a fee, businesses that pay Orbitz are going to be marginally higher cost than businesses who don't.

Those aggregators are just more gatekeepers. I prefer aggregators that perform a function for me, and are beholden to me only.

I think that way because it's the common shopping experience that consumer life is based around. It's familiar to most people.

When you shop, do you only do business with companies that sell only their own products? Or do you buy your groceries form a store that carries multiple brands of milk?

I think the writer is referring to the vacation rental industry not to hotels, which is a different ballgame altogether and where customers DO want to make comparisons and often to make decisions which are purely priced-based, for convenience. In the holiday rental business the quality of the experience for the customer should be at the top of the agenda, and this is where the aggregators have got it totally wrong, in their search for shareholder value. If you search in one of the big sites the first several pages are occupied by those properties which are going to bring the most revenue to the aggregator and which in many cases are in turn inventory provided by another site with little or no knowledge of the individual properties. You will probably never get tot he well managed properties with many 5 star reviews owned by individuals who care about your holiday experience because they are not worth as much to the aggregator so they are right at the bottom of the lists. If the aggregator cared about your holiday they would be at the top and possibly over the long term this would bring him more revenue because travellers would come to know that they could find the best properties quickly. You also cannot get to actually speak to an owner of any of the properties on the several first pages because they are all on commission rather than subscription and obviously they don't want you bypassing their system and paying the owner. Long term this is very bad for the industry and in particular for guests.

Users, specifically in this industry, often don;t find what they want. They find what the aggregator wants them to find, that's more often that not margin generated!

Then perhaps what you actually need is a better aggregator rather than denying the value of aggregators.

It's almost like what Google did that made it popular!

'a better aggregator' in what sense, exactly?

Better for the consumer would be an aggregator which ranked properties qualitatively (first page for those with clearly genuine and excellent reviews) with those which do not get such good ratings further down the list even if they are more profitable for the aggregator.

And yet the images he used in his letter came from Shutterstock, rather than directly contacting a photographer and checking they're okay with their image being used to illustrate the point he's making.

Not saying this isn't a problem, just that he doesn't like it when it brings him less money, yet he uses it in other industries because it's convenient.

Having some way to filter out the big agglomerators would be nice - obviously you can use 'buy used car -autotrader', but the majority of users don't realise they can do that, even if they would like to.

Images are good, but not needed to get his point across.

I am the author and there are no images, except a header at all in the original. They were added by TNOOZ.

Original is here: http://open-letter-to-google.blogspot.co.uk/2014/08/open-let...

Gah! SEO posts are so tedious and annoying.

Maybe if your business depends mostly on search traffic to survive it shouldn't.

Further more publicly announcing that you must be entitled to it is embarrassing, specially when the plea is a thinly veiled "you used to be better" diatribe.

If it was written as a bug report I would be more sympathetic, as it would identify specific problems, highlight desired behavior and suggest a solution, but more importantly it would be brief and to the point.

It's always fun to see Google employees admitting that yes Google only exists to extract as much wealth as possible from the rest of the world and doesn't give a single shit about anybody or anything else. If only you were able to bring the same enlightenment to some of your colleagues who post here.

The post is a bit winey, but it's a real problem. Google's search results are not much better then they were five years ago (worse in many cases). They are having a difficult time combating web-spam. Also, are you suggesting no business should try and create an ad supported website? To get the volume of viewers any public site needs organic search is a must.

"There is however one area of Google’s vast and growing empire that gives rise for serious concern and this is its foundation, “Google Search”, the most used and admired search engine on planet Earth and in fact, our solar system, unless Jupiter’s moons are hiding a quantum search tool."

If this is how he always writes then no wonder his stuff isn't ranked higher. Eschew obfuscation!

Okay. It's not Google's goal to provide the absolute best answer. They just want to avoid a wrong answer while providing an acceptable answer for a majority of searchers. Google has over-valued authority domains at the cost of smaller sites/blogs that have rich/deep/narrow content. Faking an authority domain is difficult using black-hat seo techniques so it's the sledge hammer that Google uses today. So yes, big sites are rewarded and the little guys are locked out if they try and compete.

Don't really understand his point. Just because he is a little business and hates big brands doesn't mean users in general don't want big brands preferred in their search results. If someone is searching for a music video, yes, they most often mean the big brand currently at the top of the charts. If they are searching for electronics, yes BestBuy probably gives them better results than a tiny shop. Google does track clicks on results, so they do know which results the users actually prefer.

It's basically a long-form complaint that Google doesn't show users what he wants it to.

Nope its not. It illustrates the divergence between Google mission statement based on technology and position and evoved reality.

No, it really doesn't. It illustrates a changing business reality and your fond wish that things change in a way you happen to find convenient.

You want google to provide not the thing that google and users think is most useful, but the things that YOU think is most useful. And incidentally (totally by accident, I'm sure) most profitable for you.

Who do you think has more data and is in a better position to make that call - google or you?

his point is that his business isn't ranked as high on google as he thinks it should be.

His point is that none of the businesses in his industry can get a listing within their industry. His point is that intermediate "marketplaces" inject themselves into an industry and then control consumer access to that industry's products. These "marketplaces" are not actually in these industries - they are in the internet business aggregate industry, and serve the consumers very poorly, because after all the consumer is NOT their customer, their customer is their investors.

What he utterly ignores is that to the consumer, these intermediaries are often far more valuable than any individual business in the industry is.

This is why I use Orbitz or Hipmunk to find flights, rather than going directly to United or Southwest.

>these intermediaries are often far more valuable than any individual business in the industry is.

Or they can be utter shit aggregators and middlemen, which they mostly are, and which Google spends most/all of its time on trying to defeat their SEO to get to better content. I'm pretty sure that Google understands his problem and would prefer that his site come up; what I'm not sure about is why you seem to think a problem exists if every time you search for a product, the first page is split between Best Buy, Amazon, and GetElectronicsFastAndCheap.com, and the manufacturer is relegated to the second page.

I think that Google is aware of this and trying to solve it, I just think that it's a problem that isn't possible to solve algorithmically. It is quickly leading to the end of search engine usefulness as a facilitator of discovery rather than of confirmation.

So his point is that his business isn't ranked as high on google as he thinks it should be?

His point is that Best Buy pays for that spot. If users truly wanted Best Buy at the top of their results and that happened organically, then great. That'd be great.

Instead, what has happened is that big brands are the only ones who can afford the cost of SEO, which raises the barrier to entry for everyone.

Call it Search Neutrality, if that helps get the point across.

Google doesn't owe you traffic.

I don't think Google Search's goal was to ever help the user discover an unknown business. Its to give the user the best results for their search term (which, surprise, is the more popular businesses).

Not strictly on topic, but a post like this is hard to read with so many exclamation points! Just make your dramatic point through logic, don't force it on the reader via punctuation!

I agree, Google has increasingly focused on big companies, ever since they announced the "focus on brands" 5 or so years ago.

This guy is ranking pretty good for most of the circle jerk linking scheme on the bottom of his page (which violates Google's policy - LOL if they slap him).

Reply on news.ycombinator.com