Hacker News Reader

Actually the part about search engines doesn't seem to be true... I just performed searches using Google, Yahoo, and Bing and got links to CL postings that were made within the last hour.

View on HN

They probably did it through the webmaster tools portals provided by the search engines themselves. 3Taps has gone from indexing ~2 million posts a day to virtually 0. I did a couple site:cityname.craigslist.org searches on Google, restricted to the past 24 hours, and got no results.

Google, and I presume other big search engines too, cache robots.txt for a week by default. We're well within the window for them to still be indexing CL.

If that's true, how does 3taps know that CL is blocking search engines? Also:

$ wget -q -O- --save-headers http://www.craigslist.org/robots.txt | fgrep Last-Modified

Last-Modified: Fri, 04 Nov 2011 18:13:24 GMT

Reply on news.ycombinator.com