"HOW TO CRAWL MARKETS" section has good tips for general crawling as well.