Comment by kmavm | Hacker News Reader

Comment by kmavm | Hacker News Reader

Parent Full thread

Especially in the case of spelling correction, it is not so much that Google has "solved hard problems" to get the long-tail right, as that they have a monopoly on the relevant data. Unlike all their competitors, Google has 12 years of the entire history of queries and clicks to mine for signals about how to rewrite queries. Even if you have all of Google's algorithms, it is technologically impossible to build a better query rewriter, because you don't have their data. You can't buy this data short of buying Google, and if you believe Peter Norvig, it's an irreplaceable component of Google's quality advantage.

Microsoft, and any other would-be competitor, would essentially be committing suicide not to try to make up this data gap. If their toolbar is opt-in on the part of users, and you agree with me that my click history is mine to share with Microsoft if I so choose, this is helping consumers. Without some of this data, building a viable competitor to Google is impossible, and consumers do benefit from competition in web search.

Disclaimer: I work in Facebook search. Not the same thing as web search, and I don't really care whether Bing or Google "wins", though I'm temporarily rooting for Bing because as a user I want better, more competitive web search.

> Google has 12 years of the entire history of queries and clicks to mine for signals about how to rewrite queries

How about Yahoo ? They have a longer history I suppose. IMO it's all about quality and engineering.

More importantly why did Google survived and flourished for 12 years ?

They have a longer history, but waaaay less traffic. Also, their early business model was a paid directory, not organic search results.

They have waaay less traffic only since Google came onto the scene and stole their traffic by virtue of being better.

Exactly. Peter Novig wrote a awesome spell correction algorithm.

But now Google doesnt really need ( Possibly then dont even use it) because of two set of queries where first user mispells and second where he corrects, gives them a StringMap that they can use to map mispelled queries to correct queries.

Exactly... to beat Google, you need time and data. Or you can do what MS did to fill the gap for a while.

Actually, time won't do it either. Unless you somehow compete with Google today, the data will never come.

Google only gets the query volume it does because it is the quality leader. The query volume itself helps Google to retain its quality lead. Google likes to portray search quality as being algorithm-driven, and it is to some extent, but in the modern era quality is also about collaborative filtering with clicks. If you don't have the users, you don't see the clicks, and you can't have the quality. Web search is a natural winner-take-all monopoly, unless someone gets creative, which is what Microsoft seems to have done.

      collaborative filtering with clicks ... you don't see the clicks, 
      and you can't have the quality

So why build just a "me too" product instead of finding a way to make "collaborative filtering with clicks" irrelevant?

This is so typical of Microsoft btw.

What volume do you really need, to get enough data to learn from? I'd think that 1% of Google traffic would still be a pretty big firehose to feed whatever learning algorithm you need to feed.

Don't Google, Facebook, et al run a lot of experiments for new projects on a subset of users/queries that's far smaller than 1% of traffic, and still yields very useful results?

In the case of spelling correction and query expansion, every little bit helps. Suppose you want to learn that people typing [mazad] mean [mazda]. (This is kind of a silly example, as dictionary- and edit-distance-based techniques can do corrections like this. So bear with me.) The event you need to catch is:

1. User mistakenly types a query [mazad], meaning [mazda]. (Probably less than 1% of total queries for Mazda, which is an infinitesimally tiny fraction of the total queries in your system.)

2. The user gets garbage results, and the user realizes their mistake and fixes it, rather than giving up in frustration. This is probably rather rare too, though

3. The user clicks through something that ranked highly for Mazda, and stays there long enough that your system thinks it is a "long click" that probably satisfied the user.

The golden datum here is literally a one in very-many-thousands-of-sessions event, and you need to catch a statistically meaningful number of them for every misspelling (or synonym, or whatever you're trying to learn from this data) you'd like to have your system learn. To have good coverage of the English language, we're talking about many billions of search sessions.

A previous commenter pointed out that Yahoo! probably has enough data; I bet they're right. I don't know if Yahoo! and Bing's technology partnership included access to such data.

I'm pretty sure microsoft had a search engine before Google existed...

I mostly agree with you, but I'm not sure the next frontier in web search is spelling correction. I feel like there have to be ways of producing better search results that work better than upranking the ones people click on.

Reply on news.ycombinator.com