Hacker News Reader

Comment by floatingatoll | Hacker News Reader

Dear Microsoft, case-sensitivity is important. [1]

m.bing.com/robots.txt says "/search", not "/Search". All of the crawled [2] urls are "/Search" or "/~/search".

Also,

wap.bing.com/robots.txt explicitly "Allow:"s several search pages, which are indexed by google.

[1] EDIT: Case insensitivity is often important. Above comment notes that some robots are case-insensitive. I suspect Google is not, based on the results.

[2] EDIT: I said indexed, a reply corrects to crawled. Good point, thanks.

View on HN

Small correction: All crawled urls are "/Search" or "/~/search". Note that www.bing.com/search/ is indexed (link was found on another page), but not crawled (result has no snippet).

Naturally, Bing is hosted on a Windows server, which inherits the Windows filesystem eccentricities. Case insensitivity is among those. Because "search" has six letters, that would mean that the robots.txt would need to have 64 entries to completely exclude this directory. That's not even including the tilde thing or any other paths to that directory. And that's for one directory.

Lame? Yes. Google's fault? Not in the slightest. But it brings up an interesting question: if MS clickstream gathering included an opt-out mechanism that happened to be impractical for Google, would that change the ethics of any of this? Say, by having the Bing toolbar identify itself in user-agent so that Google could block it if they wanted?

I wouldn't think that would materially change the situation. If Google really wanted to, they could probably "block" this now by encrypting their existing URL redirects, thus hiding the URL from the Bing toolbar entirely, at least until the user is out of the Google system.

What is the value of processing robots.txt in a case-sensitive way? If urls are to have different status when considering case change, the the site structure is just broken. Plus considering robots.txt in case-sensitive way has already result in lots of errors, this one included, and will result in even more in the future. Plus HTTP is not mandated to use case-sensitive URL (though it's recommended). I can't think of any argument of why robots.txt should be processed in a case-sensitive way (i mean for a good reason -- obviously search engine have a very "good" incentive to handle it that way: the possibility to cheat and index more than they should, with an excuse when they are caught), on the other hand i can think of many for case-insensitive processing...

Webmaster's shoot themselves in the foot a lot. "My site isn't showing up in google" is a frequent complaint on webmaster help forums, and typically the problem is robots.txt or meta noindex. From that standpoint, since most sites do want to be indexed, it makes sense to follow the standard as strictly as possible. It should be hard to remove your site by accident, which case insensitivity would make somewhat easier. Google states explicitly in their robots.txt policies that it is handled in a case-sensitive way.

Reply on news.ycombinator.com