I don't understand why you'd want to stop crawlers. If you didn't want people to see your content, it probably shouldn't be on the public web.
342 karma · joined March 3, 2007
presentation available here:
http://lsvp.wordpress.com/2007/04/19/presentation-at-web-20-today-show-me-the-money/