if you are crawling the entire web, you should respect robots.txt and don't fetch anything disallowed. full stop.