Hacker News Reader

1) Scrapers often rotate ip per request, rate limiting doesn’t help

Rate limiting requires them to buy more IPs (which are not free) and slows down scraping.

Yes, that’s what the big scrapers are doing (frontier AI labs). They have infinite money for this.