I am surprised people think this could affect the servers? If they scrape 300k prices from 25k pages a day, and the crawl runs every hour from 15-22 UTC, then that means the 25k is spread across 3.5k pages per crawl. Even if the crawl is aggressive and completes within a minute that is still only 58 QPS.