Hacker News Reader

Comment by adrian17 | Hacker News Reader

> After 15 minutes of confusion, it turned out Cloudflare had put a crazy robots.txt on my site without my consent (Cloudflare, love you guys, but this needs to stop).

Might be the first time I see someone complain about their website being protected from a scraper, instead of the other way around.

View on HN

I am pretty sure you have to enable cloudflare to manage your robots.txt, it shouldn't be doing that by itself. Maybe they did it by accident, it is just 1 click.

My understanding is that the current default allows AI training bots - but this actually going to change in 2 months time.

https://developers.cloudflare.com/changelog/post/2026-07-01-...

From Sept 15 all new sites added to CF will even block Googlebot by default on any page that serves ads as I understand it.

I think it's CF trying to force Google to separate out their bot traffic into bots for training and bots for the search index.

I think CF sees a big opportunity to get businesses to pay them to allow certain uses of their data but block others.

They're also starting a registry of "Approved" crawlers.

I think the issue is the lack of consent. Whether a service I use is protecting my website from scrapers or feeding everything to scrapers, some of us would prefer that it takes our informed consent before doing so.

It does.

Cloudflare is explicitly a service for dropping requests, whether it’s DDoS attacks, as a WAF, or AI crawlers. It offers a lot more too, but this isn’t Cloudflare overstepping imo.

FWIW, I just set up a domain last week, and the web UI asked if I want to block AI crawlers or not.

Perhaps OP set it up agentically, and the agent didn’t pass an optional param correctly, or ticked the box for him?

You have to enable this, or atleast was the case when I tried less than 1 month ago.

You still have to enable this as of one week ago. I suspect OP might have agentically set up a new Cloudflare domain, and who knows what the agent did during onboarding.

It should still be done with consent. Robots.txt files don’t protect against determined scrapers anyway.

I thought bot protection was one of reasons for using Cloudflare in the first place (next to general CDN hosting)? After all, they do show a captcha-like challenge on some websites, so I thought that even without robots.txt, it still would have prevented the automated request by default.

Reply on news.ycombinator.com