What "viable search engine competition" really looks like
blog.nullspace.io
blog.nullspace.io
The problem for Microsoft is the same problem that faced Apple. Being "as good as" isn't enough to get people to switch. You have to be 10 times better.
Back when Google launched, it was 10x better than Yahoo. Bing seems to have quickly gotten to 1x, but hasn't moved much further.
Pretty much: Apple was faced with a competitor who owned a vast majority of the PC market. They never took over the majority, but they managed to earn a profit from a minority of people who were willing to pay a premium for computers that worked better than the crapware offered by most Windows OEMs. Windows was cheap, and just good enough to write TPS reports.
Google now offers similar crapware: heavily-gamed and -"personalized" search results slathered with ads and just good enough to be better than most alternatives. There's plenty of room for a company to offer something better: paid subscription search, less-complete search with fewer ads and more accuracy, or something else (I'm not a "10x founder," so I can't think of these things.).
It's not like you need to buy an entirely new computer and learn to use a new operating system to switch between Bing and Google.
We're in the predictable phase where everyone hates google now because they're no longer the next big thing, but I'm not a hipster and I don't think microsoft is now the underdog so I don't get the google animosity and I don't get the sudden warmth towards microsoft.
Quite frankly I fucking hate windows, and I wish it would just go away entirely. To that end I avoid giving microsoft money at every opportunity.
If the last web site you visited was for the Audubon Society and the next thing you search for is Cardinals you will get a page about birds on the first result, but if you had just recently looked for tickets to a ball game on Ticketmaster you will get the baseball team as your first result.
One of the more fascinating things that DDG's "Dontbubbleme" campaign did, was expose just how big a swing in the results your 'meta data' has. People who are running Chrome and are "logged into Google" get very different results than people who make a query with no context following them around.
In the latter case everyone has to resort to giving you the pages that "most" people clicked on from your geographic region [1].
Bing and Google have pretty much exact equivalence if you remove tracking and geo context. (IP blind / cookie blind / search).
[1] For even more fun, set up proxy servers in various data centers around the country and do searches with no cookies or context, and proxied to different geographies. Very enlightening.
And, what if I could select my settings in a dropdown when searching. So, maybe the people at HN setup a whitelist of 100 of the best sites for programming help. I add this list to my account. Now, anytime I want to search for a programming question, I type in the query, select my custom search type from the dropdown, 'programming questions', and then click search. I'll only get results from those 100 sites.
Google will never implement this for a variety of reasons, so it's room for competition.
As the author wrote, aiming at feature parity with Google is probably a bad idea. DDG seems to do this right: they have some better features and one distinct advantage over Google with their stance on privacy. Perhaps Google will reach a certain creepiness factor with all the collected data they're factoring into search results (e.g. I have this suspicion that content from gmail is used too) and at some point it might work against them even for the average user (or am I just biased?).
One "feature" that Google has failed to do properly is custom "site search", i.e. providing a search interface for arbitrary websites. Perhaps this is another opportunity for a new competitor or even Bing (though their "site search" was shut down in 2011 apparently), there is plenty of demand (and lots of crappy custom solutions).
"When each site understands its own data better than Google, its internal search results will surpass Google’s. Google will no doubt continue to provide better global results, but the two-tiered search would decentralize efforts to improve algorithms."
I see it as one of the Achilles' heel of Google. Even if what they are doing seem impossible to copy.
- [1] Letters from the Future: Challenging Google’s Search Engine: http://blog.databigbang.com/letters-from-the-future-challeng...
This is also where dedicated "site search" providers can beat these search engines easily: they can offer a full crawl and depending on the size of the resulting index, they can bill the website owner (e.g. by storage used in GB or number of URLs).
That being said, I think the only solution to beating Google is developing something disruptive, and I mean that in a very classical and traditional sense (not the way disruptive is used to mean anything these days). It means you have to build a type of a search engine (or search solution) that Google not only has no interest in pursuing (due to conflicts of interest/opposite business incentives), but probably can't pursue, or not without employing dramatic shifts in how they do search (which again, makes it even less likely for Google to pursue and try to beat this new "disruptor").
And no, don't look at Bing to provide that. They're a "direct competitor" to Google, and they will always be, because they want to be "in the exact same business" Google is, and therefore, I don't think they'll ever radically change the way they build a search engine compared to Google.
So what can such a disruptive solution be? I can't say for sure, because it probably needs more factors/benefits on its side than just one, against Google. Social "may" be a way to disrupt Google, although I think Google is already rapidly adapting to this attack vector.
I think another could be "extreme-privacy", which Google wouldn't pursue anytime soon without really rethinking how its search business works.
Disruption also means attacking incumbent companies with much fewer resources through innovative business models and cost structures. Do something that would make Google's billions of investment in infrastructure irrelevant. A P2P search engine like YaCy or Faroo seems to be part of that solution. Perhaps what can/will kill Google is a crowdsourced super-search engine, that ends up surpassing even Google in most capabilities, because Google would be just one company, fighting against the whole resources of the world.
As in classic disruption, you also have to consider this new type of search engine will be much poorer than Google in some competition factors (at least initially). I think that is the "relevance" factor. As the op mentions, it's going to be very hard for anyone or anything to beat Google in relevance. They have too much of a headstart. But perhaps there will be such a thing as "good enough" in search, too. Maybe we're not there yet, but perhaps we will be in 5 years, or 10 years. Until a few years ago we kept wanting faster and faster Intel processors. Now we think mobile ARM processors are more than good enough for most of our daily activities.
A crowdsourced P2P search engine powered by billions of people could perhaps reach that point, too, while offering some of that "extreme privacy" and other benefits that Google can't offer.
Still, how do you get that first "10 percent" of users, that end up changing the paradigm, to start using a service that's much less relevant than Google today? You do it in the same way other disruptive technologies win out against incumbents. You promote the other benefits, that Google can't and won't match - benefits that will be important to those "first 10 percent", until you reach critical mass, and then you should be fast approaching that "good enough" relevance, too, and get even more users after that.
And that's probably because Google is incredibly weak when it comes to patents. This is kind of well-known in IP licensing circles, but it's also evident in their desperation to acquire patents (Motorola, Novell, Nortel... and other patent holding companies that nobody's never heard of).
It may not be in their nature to attack, but even if they wanted to, they are much more vulnerable to a counter-attack.
How exactly are server-side patents enforced? When the hardware on which the infringement is occurring isn't exposed to the public, how can anyone verify its presence? I know that for some (many?) of Google's search patents, there might be an indication on the client side if infringement were occurring, but I imagine that there are also many patents for which this isn't the case.
This has some problems however:
1) As you can imagine discovery may not turn up any actual infringement, making it all a huge, expensive waste of time.
2) The unfortunate reality of patent lawsuits is that if you can't prove infringement just by looking at something, you might as well have already lost. Anything that requires expert witnesses to provide input on often comes down to which sides' witness the jury finds more likable, and that's pretty much a roll of the dice.
I'm no search engine expert but I'm pretty comfortable assuming that Bing would have(and does have)had the same issues with rap genius as Google did. The only real difference is that as the market leader Google had to publicly react to rap genius 'gaming' of their system.
I think what this shows is that yes, this is precisely a product problem because if Bing were at the top we would be having this exact same discussion only with reversed roles.
Correct me if I'm wrong but Google vs Bing is not a lot different than Coke vs Pepsi.
Until we have someone come along with a search engine which is actually a better product, market share will only really be a function of effective marketing.
That said, I think your article does a good job outlining some of the barriers that any potential competitor in this arena should be aware of.
But that's not really so much of a product problem as a problem with the problem itself. In other words, it's not really clear that a better product will fix that, because the incentives will probably always point to doing this rather than not.
No one wants that (except Microsoft).
What we want is for Bing or something else to bring more balance to the search market. Today Google is so dominant that people SEO specifically for it, and Google decides what is and isn't ok to do, and can then decimate site's traffic based on that (as we saw with Rockstar in a recent example).
But if there were several successful search engines with no single dominant winner, things would be better. SEO would matter less because you would need to optimize for multiple targets, so you would get away with less tricks (or you need to work much harder to trick everyone, again with the result of less trickery). And no single company could decide the fate of every other company that needs to get traffic through web searches, which is what we have now.
I want to be "A Pepper" but DDG doesn't always give me results ... Any ideas?
Maybe. Google sells site ads; Microsoft sells Windows and Office licenses; DuckDuckGo sells search ads; Apple sells hardware. Following the money, you realize that Google's goal is to produce search results just good enough to maintain its brand, while driving as much traffic as possible to sites that display its ads. Always ask yourself how your interests differ from your search engine's.
While Microsoft's problems are real, this offers little in the way of solutions. That said; interesting read.
Do you feel something in particular sucks about the article? I'm happy to change it!
That said, I use Bing as a comparison point because it's really the only other point that's comparable in terms of, like, infrastructure and stuff. It's contextualizing more than anything. I might have removed all the references to "Bing", but I don't think it would add clarity.
Am I the only one concerned with both the fact that IE tracks its users, and the attitude of this Microsoft employee about it?
EDIT: and let's be realistic. The papers on search that come out of MSR expose way more about our infrastructure than the paltry sum presented in this post.
So there's that.
I don't, btw, thanks for asking.
No it doesn't: https://www.google.com/intl/en-US/chrome/browser/privacy/
Your search history is used (if you have it turned on), but that's just from using google.com for search and is independent of the browser you use.
The means may be different, but the ends are the same: users are tracked all across the Internet.
It's a bit confusing for me, it looks like they are collecting visited urls as they are not considered personal identifying information.
I've just learned that with the spellcheck feature they are basically receiving all the text I write on the Internet to their servers.
By the way are we really discussing about the privacy policy of a company that sent its data to NSA? I'm not condemning them, still we should be aware of this and never forget.
Nobody knows what MS is today, especially nowadays because MS doesn't have any CEO. We know that MS was the company which controlled the personal computer experience in large parts and we know that this experience was ruled by the web and by search for one decade and there is no excuse that MS totally forgot that search must be a integral part.
It has to do with the perception people have, and the fact that they more regularly associate searching the web with Google and "googling it" vs. Bing and "binging it".
At least this is the case for average, non-hacker news people.
Why does Google have this perception? I don't know, but I will guess it likely has to do with it being the one search engine that really caught on and was able to sustain its momentum.
Competing against google would not be done by any of these incremental/less important technical metrics but more having to do with changing the notion of searching the web, and how, in a likely very niched way at first.
how can 2 guys in a garage wipe out AltaVista, Lykos, and whoever else was doing search back then (with lots of smart people, data, resources, state-of-the-art software and user adoption to compete against)?
What is missing is a serious discussion on search quality (which made Google Google).
If Larry and Sergey started today, would they be able to compete against Google by providing better search results, or has search quality reached a "global maximum" of sorts?
Does search still suck?
There's also a pretty big difference in scale. The AV team working on search in 2000 wasn't all that big.
How is Bing better than Google (Search) in every way? With worse relevance and smaller index? Although, IIRC, Bing used to train their neural net based on Google search results to match the Google search quality (remember the 'torsoraphy' incident)?
> it is more important that Bing exists than OTHER CONDITION
Where OTHER CONDITION is what you've quoted. He isn't calming that OTHER CONDITION is true, merely that it isn't the most important thing.
No. Sometimes I want general information: "what is the airspeed of a laden swallow?" Sometimes I want information dependent upon the interests of the hundred or so people I know: "what do you all do in Toledo?" Sometimes I want information dependent upon my interests: "if I enjoyed 'Requiem for a Dream', what other movies should I watch?"
These things conflict: I could not possibly care less what my friends think is the airspeed of a laden swallow.
I use Google for #1, Facebook for #2, and several other sites for #3. Google has (or had) a perfectly viable business providing the best answers to general questions, and it's a shame that they're screwing it up.
Sorry, but you're wrong. Understanding a good amount of detail about your users, and in particular, about their social context, is a massive win for search relevance. Google invested quite a bit in acquiring Aardvark, for example.
So, even if you don't ask your friends about something, the information is still really important.
I think this is the crux of the issue, not that Google thinks social is the end-all answer to everything -- similarly, something like Wolfram Alpha isn't trying to solve the "which restaurant nearby should my friends and I hit tonight?" type of questions. Rather, it's more that Google wants you to be able to approach its products with ANY question or ANY search problem, and it can answer it.
Now the problem is whether or not they can pull that off, or you really need independent separate tools/services for each little question to answer.
Instead, it's the most likely approach to questions like "where can I quickly get good pizza?"
A question like that uses social search (beyond your friends!) to find people with similar taste to you, find where they eat pizza and to calculate likely travel time from your current location.
For example, reddit has a terrible search engine that users often complain about and comments aren't indexed at all. But searching with google often gives subpar results and they don't take into account things like upvotes at all. (I also think sites like reddit are punished by Google's algorithm pretty harshly.)
This might make the problem smaller and more tractable and a place where you could get an advantage over Google.
The truth behind IBM Watson is that it's based on a methodology that makes it possible to put huge teams working on different parts of the problem without tripping on each other's feet.
What makes me slightly uncomfortable are
- the name and the branding. It sounds irrational but the name annoys me and I just do not want to type it. Bing doesn't sound sophisticated, smart or anything and the big picture on the landing page is unwanted and makes the product feel unfocussed. Maybe the reason is just that Bing never did something way better than Google and thus, there were no opportunities to load this name with a good reputation with some achievements.
- the speed: Google is (or feels) still way faster here and there (when entering something into the search it instantly switch to the SERP view)
is it mostly natural language processing? matching keyword queries to document corpus
from that query how does one know user's true intent? Is he looking for a wikipedia article? is he looking to hook up? is he looking for porn? is he referring to the gender? is he just bored and trolling?
since google's reach is part of our lives, wouldn't they have a very good idea based on your previous history, what you clicked on, what search results you clicked on, what you typed in google chrome.
couldn't microsoft have pulled off the same thing?
And no MS could not have pulled off the same thing. The data MS has about you is not even close to the data GOogle has on you.
Also, users have adapted and are trained to work with Google nowdays, they memorize how to find particular results on Google (even though Google puts a lot of effort into breaking features like "link:...").
I've never cared about funnyordie, didn't even realise it still existed. Wonder how they picked that, or if other users results differ.
The good thing; when just searched for something about UIView and later type something "populating a table", it'll give me SO articles (and such) about UITableView which is what I meant but forgot to type add ios/or uitable.
The bad thing; when i'm doing research, it'll give me the same stuff over and over again and I have to page quite far and add / remove query info / play with search tools to get to different things which might not be the most popular or might not be what Google thinks I want.
The awful thing; the ads... I use ad blockers, but when I get ads (gmail / some sites), they are always the same. For Facebook I just think their algorithm is broken or maybe they have none (I only get ads about dating, cheating, stocks all of which I have no interest in nor ever shown), but for Google they show things I am interested in but I already know all of them, have been there, knew most of them before those ads and they cannot possibly sell me anything. If I wanted something from them I would go there without those ads, so why are they targeting me like that? Show me ads I don't know and if I can use it, I would click.
I'm technical, but I hear the same annoyance from others; a friend of mine rents out houses; the search results when he wants to research the competition are usually almost only his own sites (presumably because he visits them a lot?) and ads for his own properties. He is very non technical, so explaining this all to him is not really sinking in.