I would recommend hosted crawling/scrapping solutions where you don't need to dedicate your own hardware resources. Also monitoring and maintaining the scrappers/harvesters can be a pain. Shameless Plug: I work @ beevolve (www.beevolve.com) and we provide hosted crawling and scrapping solutions. PM me if you are interested.