Is there a SOTA library for common web scraping issues at scale( especially distributed over cluster of nodes) for Captcha detection, IP rotation, Rate throttling, Queue Management etc.?
In other words: Is there a drop in library to solve all the big common issues people run into scraping websites in the wild?
At least, that's how I read it.