Hacker News Reader

Comment by debacle | Hacker News Reader

Because if you ask Google to remove you from their index, they will.

Rupert Murdoch has threatened this in the past. Each time he's backed down, because he knows that being removed from Google's index is akin to being removed from the Internet.

View on HN

Which shows just how much Google profits from their virtual monopoly. If content companies knew that Google would make billions of dollars selling advertisement based on the content that they generated, it would be very hard for Google to pull the massive Internet scrapping that they managed to do from day one. It is interesting that people understand the illegitimacy of taking data from a single content source (Craigslist), but fail to see how unethical is to take data from everyone and sell advertisements based on that.

Interestingly, Craigslist is being consistent when it comes to not allowing scrapers, including Google.

http://www.craigslist.org/robots.txt

edit: oops! they're not actually blocking google from the apartment listings. thanks smackfu.

Aren't they only disallowing particular pages there? Like apartments are at *.craigslist.org/apa/number.html which doesn't seem blocked by that robots.txt.

Craigslist doesn't needs to be removed from the search index. They just need to prevent caching of posts, which is what 3Taps was abusing. Noarchive in robots.txt ought to take care of that.

To your knowledge, is it against the law to not follow a site's robots.txt? I'm not aware of one, but I am curious.

I don't know if it's illegal, but Google would follow the site's robots.txt.

Ambiguous. See http://news.ycombinator.com/item?id=4286857.

I can imagine some clever hacks where Padmapper uses Google targeted searches and Google's cache exclusively to get Craigslist data. Since Craigslist would never ask Google to remove them from their index, I wonder if Craigslist would still go after those who get their data as proxied by Google.

> Padmapper uses...Google's cache exclusively to get Craigslist data.

That's what Padmapper is doing right now. They're being sued for it.

I thought Craigslist is suing 3taps because they scrape their site directly?

Reply on news.ycombinator.com