I work a product that involves a security crawler (phish, malware detection, etc). It’s just a new arms race. Crawlers will adapt.

Cloudflare is already heavily abused by threat actors to host, and gate their malicious content. This means our crawler has to handle anti-bot and CAPTCHAs. It’s a pain. Cloudflare is no help.

They have a “verified bot” program but it’s a joke for security. You must register a unique, identifiable user agent, and come from a set of self declared IPs. Cloudflare users can check a box to filter these bots out. And now you're easily fingerprintable so the bad guys can just filter you even without Cloudflare’s help.

So now we have a choice. Operate above board and miss security threats. Or operate outside the rules (as opaquely defined by Cloudflare), and do right by our customers.

All of this on CFs side is to solve a real problem. Unfortunately by not working with the industry in a productive manner, Cloudflare is just creating new problems for everyone else.

I run an above board crawler, but in general crawler traps are relatively easy to work around, especially coming from a big target like Cloudflare, where it really pays off to build a specialized workaround that fingerprints and avoids the trap. Cloudflare's strength is arguably that they have enough traffic data they don't have to rely on stuff like this, they can gather statistics and identify bot patterns in ways smaller actors can not.

It's trickier when you have 10,000 different webmasters inventing their own solutions to do sabotage crawlers, where the juice isn't worth the squeeze when it comes to implementing individual workarounds.

This won't stop the big ones, Google, Meta, OpenAI, Perplexity, or even the Chinese Govt. But it will make it harder for new entrants.

I am not involved in scraping, but to me this sounds like simply another tool in the arsenal. They say it's hard for the scraper to realize it has been caught this way because it's not being blocked. However, I don't see anything preventing scrapers from implementing heuristics to realize that.