I found out, through reddit a couple of days ago, about Pomp:

https://bitbucket.org/estin/pomp

It looks like a much cleaner Scrapy-inspired spider framework, without the twisted dependency. And it's python 2+3 compatible. I'm very excited to try it out.

I love that the Python community is coming up with so many robust frameworks. I have used Scrapy, and it provides most boilerplate functionality out of the box. Gone are the days I would use wget for my scraping tasks. This being said, it's a bit disappointing that Scrapy still doesn't have native support for dynamic pages. I am hoping to see this feature in the upcoming releases. With more and more of the web becoming dynamic, this should be priority feature. Other than that, I have nothing but praise for Scrapy. Pomp is apparently a simplistic take on Scraping, but it doesn't handle redirects, caching, cookies, authentication etc. I wonder if it provides parallel processing out of the box; most likely not. This is a bit strange, because these are the features for which I would prefer using a framework over writing my own code, which makes me less inclined to try Pomp. Scrapy for the win :D

Just seeing this for the first time. Got a proof-of-concept demo working in no time (after fussing with install requirements...). Looks like a great tool for non-devs like me who still need to scrape things occasionally for data collection and analysis.

My vote goes to pyspider[1]

[1] - https://github.com/binux/pyspider

Another vote for scrapy, massively useful for web crawling. Particularly love the shell for debugging things.

No Python 3 support yet?