Craigslist reportedly blocks search engines to kill competitors

arstechnica.com

14 points·rakingleaves··5 comments

As a PadMapper user this saddens me, but I can only wonder why they didn't do it sooner. Craigslist posts are so short-lived, I don't know why they would want them indexed.

Did these people do any fact checking? I just ran another test:

visit http://sfbay.craigslist.org/bia/

spot ad for a kids Giant MTX 250 bike http://sfbay.craigslist.org/nby/bik/3189980073.html

perform a google search for the same thing http://www.google.com/search?q=site%3Acraigslist.org+giant+m...

result number two in the set is that posting, with the following summary:

Giant MTX 250 Kid's Mountain Bike sfbay.craigslist.org/nby/bik/3189980073.html 4 minutes ago ... Giant kid sized hardtail mountain bike for sale. It is a 21 speed, and it has front suspension. The bike was always taken care of, and is in perfect ...

"4 minutes ago" tells me Google is having no problems hitting craigslist.

...Do you do any fact checking?

I just checked the source:

Nothing new. That tag has always been in place. It's to prevent using an ad for SEO purposes

Those tags aren't as obvious in interpretation as they first appear. A full write-up is a http://noarchive.net/meta/ for completeness, but in short:

NOARCHIVE: ok to index this page, but don't cache the result permanently

NOFOLLOW: don't follow any links on this page

Note that these two things don't prevent a search engine from visiting a page, indexing it, and providing search results for it.

Reply on news.ycombinator.com