> status: complete (28.284 seconds / 14 results)
Why was it so slow? Is there a default delay between requests or something?
Why was it so slow? Is there a default delay between requests or something?
We found that it becomes too easy to break API request limits or spam a website otherwise.
However if you rerun that query it should load pretty instantly due to the caching layers, so the actual querying/filtering of the data part is smoother/faster.
Every time I see that, the "2 hardest things" springs to mind. Is there a clear-caches option, or I guess the opposite question: does that process honor the HTTP caching semantics? Scrapy actually has a bunch of configurable knobs for that (use RFC2616 Policy ( https://docs.scrapy.org/en/2.8/topics/downloader-middleware.... ), write your own policy, or a ton of other stuff: https://docs.scrapy.org/en/2.8/topics/downloader-middleware.... )
Your provided links are interesting and something for us think about some more. Honestly, I would be quite interested in hearing more about your experiences.
Wait, so we literally can't go faster than 1 req/s unless we pay?
I have to say I'm pretty disappointed :/
Don't spam APIs. That said, if you're determined to do so, there's not much this or any other tool can do to stop you from trying.
Scheduling and Domain Policies were the main features we chose to gate initially as they don't affect core functionality other than performance and deployment.