For a particular type of scraping, we wrote SSScraper on top of Colly and it works really well:

https://github.com/gotripod/ssscraper/