Bad guys might download the robots.txt to find out the stuff they don't want them to crawl.