Hacker News Reader

How much benefit does the spidering/progress/pause/resume functionality give if you're not just spidering every link on the site, but have complex logic to determine exactly which links to crawl and in what order? Does Scrapy provide convenient extension hooks to change the crawl algorithm?

View on HN

I haven't had the need to use pause/resume but I do incorporate logic (not necessarily all that complex) in determining which links to crawl. It is very easy to do within each spider especially with how the framework uses generators. It is also easy to extend the pipelines for pre and post processing.

As others have said, managing a project with Scrapy is super easy and highly configurable with sane out of the box settings.

I haven't done this, but take a peek at these. Perhaps you can have your own linkextractor that rejects certain items, or perhaps a custom scheduler:

https://doc.scrapy.org/en/latest/topics/link-extractors.html https://doc.scrapy.org/en/0.14/topics/architecture.html?high... https://doc.scrapy.org/en/0.9/experimental/scheduler-middlew... https://doc.scrapy.org/en/latest/topics/settings.html#schedu... https://scrapy.readthedocs.io/en/0.14/faq.html#does-scrapy-c...

Reply on news.ycombinator.com