I think that Facebook has a right to protect their data, but leaving robots.txt wide open and trying to enforce some random TOS that a developer might not even know about is the wrong way to do it. I think a much better method would be to ban all bots from robots.txt (and crawling) except the major search engines and provide data to others through APIs or other means that are controlled (and the TOS can apply to the API rather than random crawling). Maybe not what a developer wants to hear, but the truth is that the internet is infested with bots that do nefarious things, and I think it's better for the privacy and security of their userbase to control the spread of their data as much as possible.