Hacker News Reader

Comment by Lxr | Hacker News Reader

I have done a lot of scraping in Python with requests and lxml and never really understood what scrapy offers beyond that. What are the main features that can't be easily implemented manually?

View on HN

Pluggable parsers, automatically good error handling and spidering functionality (finding and queueing new links to scrape), great logging, progress stats, exports, pause/resume functionality, and a million other goodies that are seemingly "trivial" but really you don't want to rewrite them every time you write a scraper.

edit: Especially if your scraping jobs take a LONG time - days and weeks, this stuff is extra handy. Might I add a great debugging environment (scrapy shell), error handling, rate limiting, respecting robots.txt, so much more.

How much benefit does the spidering/progress/pause/resume functionality give if you're not just spidering every link on the site, but have complex logic to determine exactly which links to crawl and in what order? Does Scrapy provide convenient extension hooks to change the crawl algorithm?

I haven't had the need to use pause/resume but I do incorporate logic (not necessarily all that complex) in determining which links to crawl. It is very easy to do within each spider especially with how the framework uses generators. It is also easy to extend the pipelines for pre and post processing.

As others have said, managing a project with Scrapy is super easy and highly configurable with sane out of the box settings.

I haven't done this, but take a peek at these. Perhaps you can have your own linkextractor that rejects certain items, or perhaps a custom scheduler:

https://doc.scrapy.org/en/latest/topics/link-extractors.html https://doc.scrapy.org/en/0.14/topics/architecture.html?high... https://doc.scrapy.org/en/0.9/experimental/scheduler-middlew... https://doc.scrapy.org/en/latest/topics/settings.html#schedu... https://scrapy.readthedocs.io/en/0.14/faq.html#does-scrapy-c...

I tried scrapy many years ago when it was pretty new. I felt like it mostly just got in the way back then and resorted to normal HTTP lib scraping methods like you've described until recently, when I decided to give scrapy another try.

I really enjoyed it this time. It has mostly figured out how to be unobtrusive and it now provides a lot of handy stuff out of the box, like the scrapy shell, the ability to easily retry certain error codes, a built-in caching mechanism so that multiple iterative runs are semi-reasonable, and an intelligent crawler that automatically ignores duplicate links from the same session. Compatibility with things like Scrapy Cloud is a nice bonus.

It's worth a try if you haven't looked at it in a while.

The Scrapy cloud alone is worth it. And it gives guidance, making the start very easy.

Reply on news.ycombinator.com