Hacker News Reader

>We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time.

What's the point of this compared to letting cloudflare handle everything automagically? Presumably whatever heuristics they come up with are going to be better than you can, given limited time and budget?

View on HN

One, their functionality is bad, two, we don't want to block robot traffic. We're redicting it to a separate sever so that overly-large waves don't bring down our main servers. We definitely want to be scraped.

To your point: I think you're simply mistaken that they offer a good scraper-detection solution, but I'm all ears.

Reply on news.ycombinator.com