Hacker News Reader

Comment by wickedchicken | Hacker News Reader

I see this from the opposite viewpoint, that PadMapper is operating as a search engine designed to help you find the right Craigslist posts for your apartment hunt. Claiming that PadMapper's success is from Craigslist failing to 'provide a good UI' is like suggesting that Google News' success is because newspaper companies failed to provide good article SEO. Instead, it's a different modality of content discovery, and it ultimately redirects you to the original source.

In my (personal) opinion, I would argue that the content providers or aggregators and search engines benefit synergistically. CL and Belgian newspapers appear to disagree.

View on HN

I don't think this has any legal bearing for anyone, but it's worth noting that Google respects publishers who do not wish to be indexed.

Here's the abstract for a nice review of search engine law: http://works.bepress.com/james_grimmelmann/13/ If I remember correctly, indexing a site that asks to not be indexed might be illegal as an illegal tresspass, but it is not settled law. The argument is that you are stealing resources (computer time) from the site owner.

>The argument is that you are stealing resources (computer time) from the site owner.

That's why 3taps is getting the data from google's cache without touching craigslist servers.

And makes it impossible to block them as a side effect.

It's not impossible. You just tell google not to cache.

How the heck are they scraping Google without being banned or rate limited?

There are a lot of companies that are scraping Google quite successfully. Many of these are for 'rank checking' services that provide ranking data for certain keywords over time; these are heavily used by SEO and marketing agencies.

The two that jump to mind are Authority Labs and SEOmoz.

I guess: a shed load of proxies. :)

Amazon/ other clouds out there. Just auto provision your instances (lots of them), scrap, sleep, wake, scrap, sleep...

IANAL but AFAIK it is only a civil matter (i.e. not illegal) since it is a usually prosecuted as a tort of trespass to chattels. For such a case to succeed the prosecution needs to show that the actions of the defendant deprived them of use of the good they were trespassing on. i.e. they need to cause enough of a burden on the servers that the claimant or their customers could not use the service.

What I'd care more about is whether the people posting the ads wish to be indexed; I'd be surprised if many don't. (I don't know about the legal bearing either.)

Perhaps craigslist could put up a checkbox (like they do with the never-checked "It's okay to contact me about products ...") that says something like:

"I'm okay with people finding this listing through another service."

Perhaps because they don't want people to get to their listings that way even if they want to, and thus don't want your opinion? (Edit: and perhaps they really aren't as interested as they claim in making it easier for buyers and sellers to find each other?)

So eventually this comes down to CL trying to protect it's monopoly as the listing site of the internet

For that analogy to hold, PadMapper would have to be pulling data from a large number of sources. The value that Google News provides is that it aggregates a very large number of sources. PadMapper does not.

Padmapper collates data from multiple sources: Craigslist, Rent.com, Apartments.com, and "Others" (not sure which those would be).

It's pretty clear PadMapper was BUILT off of Craigslist that CL's poor UI is isn't entire reason for being, and it never would have been successful without CL. It's disingenuous to ignore that by listing "rent.com" as an alternate source.

> PadMapper does not.

It does.

But without the Craigslist data it has no value.

Total nonsense. I found my current place with them, without craigslist.

It does not. One main source and a handful of secondary sources is not at all comparable to a true aggregator like Google News.

Look, it's obvious that even PadMapper believes Craigslist is THE data source they must have.

Google News could drop any given data source and not blink. PadMapper simply is not in the same position.

In my city, Craigslist is hardly used for rental listings. Everybody uses kijiji.com, which is available on Padmapper. There are alternative data sources, and they are actually used. Just not where you are.

But that is where the source of the problem is - CL has the largest user base for listing and classifieds on the internet and since they have the monopoly on this they have never felt the need to improve the experience on their site. Now that padmapper has started getting large number of users CL feels threatened and is suing them to shut them down.

Does PadMapper respect the robots.txt file?

PadMapper uses 3Taps.com to get the data. 3Taps gets Craigslist data through Google. Google's bot respects robots.txt.

So, yes.

[edit for clarity, and for misspelling clarity]

Well... "sort of", then. PadMapper respects the robots.txt for GoogleBot, not for PadMapper.

Presumably 3Taps violates Google's ToS to get the data from them ?

I would really love to know that kind of shady proxy bot army they have implemented in order to scrape Google on such a scale.

This type of comment is common. Not only on HN but on forums in general.

While I agree the web is full of lowlifes engaged in web development, many of them in porn or some other area that appeals to base instincts, I find this comment perplexing. Because it is so subjective, yet it tries to seem objective by focusing on some random criteria.

Google employs a "bot army" to scrape the entire web. So what?

If the comment was something like "I don't like Company X." Or even "I don't like Company X because...", it would make sense to me.

But that is not how this common type of comment goes. Instead it suggests that bot=evil, i.e. any sort of automation or any sort of data collection by anyone other than [your favorite company] is "shady".

That's crazy. IMO.

It's what a company does with the data that matters.

Anyway, I'm not keen on 3Taps because they are not provinding bulk data, only API's that require "developer keys". Why?

Either you are going to democratise data, or you are just another schemer trying to find ways to collect infromation about people, in this case people using "your API".

I don't want API's I want the data. I can make my own interfaces thank you.

I thought it was weird that CL didn't add a Disallow line for padmapper to robots.txt from the start (just from a PR perspective).

But robots.txt has no special legal authority, it's just a convention used to communicate a publisher's intent. I'm pretty sure the C&D letter made it 100% clear that CL did not want Padmapper crawling their site or using their data.

Padmapper doesn't crawl Craigslist. That's not how it happens.

...any more. Now they are using a third party, but at the time Craigslist sent the C&D they were scraping the site directly.

I know, but I thought the existence of robots.txt was why Google is allowed to crawl sites. If a site disagrees with the crawling they can add a robots.txt entry and Google will honor it. It at least shows that you are giving the publisher an option.

Reply on news.ycombinator.com