Comment by InclinedPlane | Hacker News Reader

Comment by InclinedPlane | Hacker News Reader

Here's the problem in a nutshell:

The Bing toolbar uses clickstream data to extract information about the relatedness of urls, just like any search engine crawler does (page rank works this way). When a user clicks on url B from url A the Bing toolbar sends information back to MS about the relatedness of urls A and B including all the meta-data involved. If url A is ".../search?q=torsorophy" then Bing will make a note of that and will start showing results for B when you search for "torsorophy".

In principle this isn't necessarily a bad thing, it allows Bing to index sites that it wouldn't otherwise, letting its search index grow more organically. However, when search engines come into play things get problematic, because now the Bing toolbar is little more than an automated method for scraping search results piecemeal. Given that a great deal of modern web surfing falls into the "search for X, click links for X" pattern, this should have been something that the Bing engineers anticipated (if they didn't anticipate it that's bad enough, if they did and ignored the problem that's much worse).

Worse yet, the Bing toolbar is effectively a search indexer which does not respect robots.txt. Let that sink in for a while. Google.com/robots.txt has this line: "Disallow: /search", and yet apparently the Bing toolbar has absolutely no compunction about effectively ignoring that.

tl;dr: MS has created a search indexer which ignores robots.txt, this is bad.

Yours is the first explanation I've seen that makes any case for Microsoft not being intentionally cheating. All the responses I've seen from Microsoft are in the form "we're not copying Google we're just using user click information" which is poor since what Google showed is that they're associating click information with the search terms that were put into Google. But since those search terms are in the referrer URL it could conceivably be an innocent general algorithm weighing the words in the URL.

As for ignoring robots.txt, that may not be the case. Conceivably you could get the url A->B link, save the metadata for both, signaling them as related, and then check both URLs against robots.txt to see if you should have them in the index. Then if url A is ".../search?q=torsorophy" Google's robots.txt disallows it from being indexed and only url B gets in but the link to "torsorophy" is still there from the metadata.

Indeed. Certainly it is technologically possible for clickstream based indexing to still abide by robots.txt rules. However, the Bing toolbar does not. That is the key issue here.

How do you know it does not? I'm assuming robots.txt is about preventing the page contents from being crawled and added to the index. If all they use the click info for is to associate referrers (google URLs in this case) to pages in the index and they don't crawl the google search itself I don't see how that breaks the robots.txt contract.

The page contents are being crawled and added to the index, but by Bing Toolbar users, not a computer program. I consider that to be an underhanded way to circumvent robots.txt, but others might not.

What makes you say that? We haven't seen anything to indicate Google search pages are in the Bing index.

In fact what Google users are really clicking in searches are "google.com/url?" URL's which are also disallowed in robots.txt(while the url they redirect to aren't).

"...the Bing toolbar is effectively a search indexer which does not respect robots.txt..."

maybe i don't understand, but your logic seems correct until the above statement.

suppose you had a system that only used clickstream data. so you store a big list of url pairs (A,B) and a probability that B will follow A. i believe that your argument relies on the fact that it's possible for this system to violate a robots.txt file and i don't yet see it.

The intention of robots.txt is to tell search systems specifically "do not use information from the following pages in building a search index". The Bing toolbar's use of clickstream data from google, and no doubt many other sites, clearly violates that spirit.

This could easily be fixed, by checking the clickstream data against robots.txt files and discarding data that shouldn't be used. Microsoft apparently has decided not to take that step.

your assumptions:

  - the "intention" of the robots.txt standard is as you state
  - the url is included in the information not allowed by that standard
  - if the url is not included it should be because of the "intention"
  - toolbars are subject to the same standards

I'm not disagreeing with you as much as just pointing out that I don't think EVERYONE agrees on these standards.

Err, robots.txt doesn't even enter into the picture. It would matter only if Bing's server was requesting pages off Google's site. That is not happening. The only things that are going are the Google URL and only the clicked URL(not the other Google results that the user didn't click on).

The user gave permission to Microsoft to use this info by installing the toolbar. I don't see why or how the Bing tool bar should visit Google.com/robots.txt to see the blocked folders unless they were crawling Google pages.

But it's the perfect way to circumvent robots.txt! With enough users the law of large numbers applies (and how many hundreds of millions of users are using IE?) so you can a.s. copy Google's index.

In particular, you copy the most popular results, so if you weight the % of google's index that you copied, then it's going to be a huge number, and almost for free.

It's evil. But devilishly smart.

The contention you're making, and apparently Microsoft as well, is that the Bing toolbar is not a search indexer and does not need to obey the conventions of robots.txt?

Personally I don't think that flies. Should I be able to create a toolbar that rips the data from hulu.com and hosts it elsewhere?

this is not my area of expertise but i really don't think that's a proper analogy. a more relevant phrasing would be, "should i be able to create a toolbar that inspects (previously/explicitly) requested documents from hulu for external links". and i would be inclined to say, "yes"...

Reply on news.ycombinator.com