Hacker News Reader

Comment by eel | Hacker News Reader

It's not against the robots.txt rules. In Facebook's robots.txt file, there is a section for the Googlebot (as well as several other bots)

  User-agent: Googlebot
  Disallow: /ac.php
  ...

A bot is supposed to follow the its own User-agent section. If that is not found, it should follow the User-agent: * section. [1]

[1] http://www.robotstxt.org/orig.html

View on HN

Great Reply, thanks.

Any supporting link reference for this ?

I edited the original comment, but yes, it is in this document: http://www.robotstxt.org/orig.html

The key paragraph is:

  If the value is '*', the record describes the default
  access policy for any robot that has not matched any of
  the other records. It is not allowed to have multiple
  such records in the "/robots.txt" file.

Thanks for sharing this !

Reply on news.ycombinator.com