Craigslist is delivering exactly what it its users signed up for (no more no less). People posting ads on Craigslist do not necessarily want or intend for it to be reposted on other sites.
Craigslist is delivering exactly what it its users signed up for (no more no less). People posting ads on Craigslist do not necessarily want or intend for it to be reposted on other sites.
That argument appears to be invalidated by Craiglist's own terms of use, which say that when you upload a listing to Craigslist they can syndicate it wherever they want (http://news.ycombinator.com/item?id=4287519).
Still, I think there's something admirable in the simplicity and transparency of interacting with Craigslist. What you see is pretty much exactly what you get.
I had to giggle a little bit at this in the grand scheme of the internet, sorry.
When I posted something on Facebook Marketplace, I started getting emails and comments from other "market" sites that Facebook had cross-posted my listing to.
While it was annoying to not know this up front, the fact that it was more visible and getting more bites because of it only helped me make the sale quicker. If a service wants to piggyback off of another to make my postings more buoyant, as a user and seller, I don't have a problem with it.
If this is illegal or otherwise objectionable without an explicit agreement from each Craigslist poster, I'm not really sure how a search engine or even descriptive hyperlinking is kosher without explicit agreement from each website indexed or referred to.
The point is Google respects publishers' desire not to be indexed. Padmapper does not. Robots.txt is simply a common method for conveying that message. It's not like Padmapper could argue they didn't know CL was unhappy; they got a certified letter!
This is just semantics. Google respects publishers who do not want their sites listed.