Hacker News Reader

Comment by dr42 | Hacker News Reader

The difference is that the search engines aggregate from the whole web, PadMapper was just appropriating content from CL, which is against their TOS. It's very clearly spelled out.

All the other companies that have tried have also been told to C&D. If you don't like it, start your own network...

From http://www.craigslist.org/about/terms.of.use

"Any copying, aggregation, display, distribution, performance or derivative use of craigslist or any content posted on craigslist whether done directly or through intermediaries (including but not limited to by means of spiders, robots, crawlers, scrapers, framing, iframes or RSS feeds) is prohibited."

View on HN

PadMapper gets content from more than just Craigslist. They have their own postings (padlister.com), sublet.com, apartments.com, apartmentfinder.com and a lot more. There's really not that much difference between them and a search engine.

I've heard this argument many times before, it was the same with Oodle, but PadMapper's business model is to aggregate apartment listings (Google's is not) and this is in contradiction to the TOS so inevitably Craig Newmark sends them a C&D letter.

Several innovative solutions have been invented over the years, the most obvious one is to do the CL scraping on the client, this way there is no indexing server to block, it's just the client reading all the search listings, parsing them and offering faceted filters. It breaks the spirit of the TOS but it would be impossible for CL to block. I am not advocating this, just pointing out one of the many approaches that have been attempted over the years.

Not true. Padmapper has multiple DBs.

Google can scan apartment listings and show them as results in their format. PadMapper cant. What's the difference?

See http://www.craigslist.org/robots.txt .

I'm not sure why those TOUs would prohibit Padmapper but not Google.

The immediately following text in the CL ToS reads:

As a limited exception, general purpose Internet search engines and noncommercial public archives will be entitled to access craigslist without individual written agreements executed with CL that specifically authorize an exception to this prohibition if, in all cases and individual instances: (a) they provide a direct hyperlink to the relevant craigslist website, service, forum or content; (b) they access craigslist from a stable IP address using an easily identifiable agent; and (c) they comply with CL's robots.txt file ...

Reply on news.ycombinator.com