Hacker News Reader

CL's robot.txt file does not 'disallow' crawling the real estate section. So Padmapper was likely as compliant as Google.

http://www.craigslist.org/robots.txt

No, this is just semantics.

The point is Google respects publishers' desire not to be indexed. Padmapper does not. Robots.txt is simply a common method for conveying that message. It's not like Padmapper could argue they didn't know CL was unhappy; they got a certified letter!

This is just semantics. Google respects publishers who do not want their sites listed.

Good luck explaining that to a judge. Would you gamble a company based on that distinction?

True, but that's irrelevant to the comment I was replying to. We are talking about user expectations with regard to content they posted being made available on other sites.