Scrapinghub/Scrapyrt: Scrapy Realtime

github.com

18 points·r_singh··4 comments

I'm using scrapinghub extensively for https://pagewatch.dev , is this project something you can use as a self-hosted replacement? Its not very clear how it works, what does the Realtime mean?

what does the Realtime mean?

I believe they are contrasting it with how something like https://github.com/scrapy/scrapyd#readme or the ubiquitous "run scrapy under crontab" setups normally function: you kick off the spider, then you go get coffee and wait for the ItemPipeline to deliver the results somewhere else

With scrapyrt, you submit a POST or GET, and that URL blocks until the one requested thing completes or times out, enabling (only conceptually, mind you) an archive.is replacement using an XHR

Looks to just put an API in place. I think they are using creative license with the term "realtime" (since I presume all scraping actions are queued and async)

and async

I don't get that impression from the description of the response from `POST`: https://scrapyrt.readthedocs.io/en/0.11.0/api.html#success-r... since it does not return a job-id that requires polling, it appears to block until your scrape request is completed or it times out

It's likely an implementation detail about whether the scrapyrt component is the one which blocks, but otherwise uses queuing and asynchronous invocations when interacting with Scrapy, or whether -- as the custom CrawlManager implies (https://scrapyrt.readthedocs.io/en/0.11.0/api.html#crawl-man...) -- scrapyrt actually takes over and makes the entire CrawlManager -> Scheduler -> Spider call stack synchronous and thus able to respond to a POST within https://scrapyrt.readthedocs.io/en/0.11.0/api.html#timeout-l... seconds with the actual Items

Reply on news.ycombinator.com