Agreed (especially on the devops overhead), some options to lower the bandwidth usage using request blocking: - block ads - blocks videos, images - css stylesheets

In the past I've built scraper infrastructure (headless pools, credential stores, proxy managers, agent profiles) and managed to get a pretty efficient service by tracking cpu,memory,network usage per each job and writing specialized versions. I got pretty far trying to automatically generate specialized scrapers from previous requests but I moved to other projects.

Scraping becomes boring really fast if you don't use the data in meaningful ways.