>
I've spent about 1 month to test how scrapy could be a fit for our purposes. And, quite surprisingly, it was hard to design a distributed web crawler. Scrapy is great for those in-the-middle tasks where you need to crawl a bit + process data on the go.I agree (but in my case I needed 3 months to understand that :) ).
I did start with plain "Scrapy" and it did work fine (maybe some parameters could be explained a bit better especially the ones involving concurrency) and it has been in general very helpful to initially understand some concepts related to web scraping.
But now, after having rewritten multiple parts like 50 times to be able to handle most of what can happen, having split the workload into a multi-staged pipeline involving different specialized programs, having implemented custom code to distribute domains to be scraped to dedicated subprocesses (to greatly improve performance), and much more... I am using Scrapy basically to only download the raw bytes and to take care of what's mentioned in "robots.txt" => my next step will most probably be to replace it (now I don't like a lot having it as a dependency for my program, I therefore started thinking that such a complicated lib is a bit a "risk") with "requests" and something that can help to interpret "robots.txt".
In my case ("wide" Internet scan involving so far ~8M webpages distributed over ~500K domains) I think that it might have been better to start straight with just using "requests".