Hacker News Reader

What's the penalty (if any) for ignoring robots.txt and crawling anyway? Any citations of it being legally enforced via TOS or ASP?

View on HN

robots.txt in the past has been the legal defense against claims of copyright infringement.

Curious if you have any citations for this remark. I have always been under the impression that robots.txt is entirely optional, both to implement and to respect.

IANAL and looking for other sources, but it's discussed along with many other factors here: http://www.benedict.com/Digital/Internet/Field/Field.aspx

Essentially, the existence of robots.txt and meta noindex has made courts more comfortable ruling that the index of search engines constitutes fair use.

But this still doesn't specifically answer the question, "What is the penalty for ignoring 'User-agent:* Disallow: /' in robots.txt, or meta tag no-index", only that without a robots.txt or no-index tag, crawlers can't be found liable for accessing information that is freely available to them.

Interesting link, however. I was unaware of that case.

There is no penalty, but they can always block the crawler or serve rubbish.

Reply on news.ycombinator.com