maybe it has been discussed before (i didn't follow the craigslist discussions) but it's funny that craigslist does not have any robots.txt on it's city subdomains

i.e.: http://newyork.craigslist.org/mnh/sub/3195940666.html is hosted on http://newyork.craigslist.org/

but http://newyork.craigslist.org/robots.txt makes an HTTP 302 redirect to http://www.craigslist.org/robots.txt, but http://www.craigslist.org/robots.txt is only a valid robots.txt for http://www.craigslist.org, and not for http://newyork.craigslist.org/, as the host, protocol and port of the robots.txt file is an essential part of its validity range.

so basically newyork.craigslist.org does not have any robots.txt and can be crawled as you like.

if you want more about it, see https://developers.google.com/webmasters/control-crawl-index...

  >The directives listed in the robots.txt file apply only to 
  >the host, protocol and port number where the file is hosted.