Craigslist reportedly removes listings from search engines

theverge.com

15 points·protomyth··5 comments

Isn't the way for Craigslist to remove its posts from search results via http://www.craigslist.org/robots.txt?

If you take a look at a page like http://portland.craigslist.org/mlt/eng/3189881249.html

You can see that they added a <meta name="robots" content="NOARCHIVE,NOFOLLOW"> tag. So Google will index it and show it, but it won't cache the full text. This change means that companies like 3taps can no longer get the full text from the Google cache, while still allowing search results to show in searches.

http://www.seroundtable.com/archives/019533.html

That article is from 2009, noting that CL had added the meta tag to posting pages. So I think this is old news.

I think storborg has it right in http://news.ycombinator.com/item?id=4353120 when saying that "3taps seems to be claiming that Craigslist has cut off Google, but I think it's just that Craigslist has cut off 3taps."

am I the only one who uses craiggers? although I suppose they'll have to go the way of the dodo soon enough now

I just tried the following:

visited http://sfbay.craigslist.org/bia/

saw ad for "Burning Man Beach Cruiser Bicycle" http://sfbay.craigslist.org/eby/bik/3189652975.html

performed the following search on google http://www.google.com/search?q=site%3Acraigslist.org+neon+gr...

first result returned is that ad, with the following snippet: Burning Man Beach Cruiser Bicycle sfbay.craigslist.org/eby/bik/3189652975.html 22 minutes ago ... Neon Green cruiser bicycle, perfect for the playa, or cruising around town. Single speed, great shape. Available nights and weekends.

"22 minutes ago" sounds like Google is still actively crawling content from craigslist to me.

Reply on news.ycombinator.com