We wholeheartedly support this cause at Diffbot! http://www.diffbot.com/robots.txt