Hacker News Reader

Comment by Aaronontheweb | Hacker News Reader

Pete's right-on about this: "You've chosen to leave all that information out in the open so you can benefit from the search traffic, and instead try to change the established rules of the web so you can selectively sue anyone you decide is a threat. "

Speaking as someone who's working on leveraging the Facebook API in a commercial product, this leaves me feeling like I'm opening myself to a lot of legal exposure if Facebook subjectively decides that my service poses even a minor threat to them. Given that I'm bootstrapping, there's no way I'd be able to put up any sort of legal fight what so ever against a company as well-funded as Facebook.

View on HN

I think that Facebook has a right to protect their data, but leaving robots.txt wide open and trying to enforce some random TOS that a developer might not even know about is the wrong way to do it. I think a much better method would be to ban all bots from robots.txt (and crawling) except the major search engines and provide data to others through APIs or other means that are controlled (and the TOS can apply to the API rather than random crawling). Maybe not what a developer wants to hear, but the truth is that the internet is infested with bots that do nefarious things, and I think it's better for the privacy and security of their userbase to control the spread of their data as much as possible.

And I absolutely agree with you - it's the subjectivity in their choice of whether or not something actually violates their policies which gives me pause. If you can't figure out in good faith if you're actually complying with their rules or not then that's a problem for developers, myself and Pete included.

I am unfamiliar with the details of a robots.txt file. Is it possible to specify "I only want bots x, y and z to crawl my site"?

yes, using the user-agent:

http://en.wikipedia.org/wiki/Robots_exclusion_standard

How difficult is it for a bot to lie?

Though if you've made it clear that only x, y and z can crawl your site, and someone spoofs, say, y, then it would be easy to demonstrate that someone has done something they know they shouldn't.

incredibly easy.

and not only can the bot lie, it can disregard the robots.txt file altogether. just like the terms of service document for humans, you can choose to disregard it & deal w/ the consequences (blocked IP's, lawsuit, etc).

robots.txt is just a version of the TOS that computers can read.

They could also make the pages NOT indexable and then have a side agreement with Google to give Google API access to index the pages they want.

What? So you want to just make Google more powerful? How about all the other search engines, even the less known ones like Duck Duck Go?

Robots.txt governs automated retrieval of a page's contents. I don't think it makes any comments about what the retriever does with that data. Those uses are governed by other sets of rules. For instance, just because a site's robots.txt invites automated retrieval doesn't make it a legitimate use to just re-host that content elsewhere as your own.

I think Facebook's ToS is an appropriate way for them to send messages about what they'll do if you make certain uses of the content you retrieve from their site. However, these ToS documents aren't omnipotent: they can't restrict some fair uses to which you might put the data, for instance, or bind you to silly terms. IANAL, but I think the ToS is probably best understood as an intent to use still other sets of rules (perhaps selectively) if you do certain things with retrieved data. Disagreeing with that policy is certainly possible, but I don't think robots.txt has much to do with it.

Reply on news.ycombinator.com