Hacker News Reader

I would hope that google, of all companies, would be capable of writing a crawler that ignores robots.txt comments.

Other than that the guy only has one rule.

Anybody know how big robots.txt can be and if anyone has ever given a spider grief by screwing with the size or download rate of robots.txt?

The robots.txt limit in Google's Webmasterpanelthing is 100,000 characters. Googlebot has been said to stop parsing robots.txt after 500,000 characters. I don't know if that's bytes or characters.

500kb is the hard upper limit in the crawler itself. Limits that are exposed elsewhere (Webmaster Tools, etc) are irrelevant.

I bet they are fairly liberal with their size limits:

http://www.amazon.com/robots.txt http://aws.amazon.com/robots.txt http://www.google.com/robots.txt http://store.apple.com/robots.txt

Apple should use git instead of subversion :)

robots this size aren't that unusual http://www.bbc.co.uk/robots.txt

wow, just realized this is a great way to find information about websites

I think you just proved the point here: SEO can be very valuable. Any good SEO would point out to the BBC that having similar rules for User-agent: Googlebot-Mobile and User-agent: * is not very efficient.