Hacker News Reader

Hacker News

Comment by rzz3 | Hacker News Reader

Parent Full thread

rzz3

[flagged]

View on HN

> No, he (Matthew) opted everyone in by default

Now you're just lying.

I checked several of my Cloudflare sites and none have it enabled by default:

"No robots.txt file found. Consider enabling Cloudflare managed robots.txt or generate one for your website"

"A robots.txt was found and is not managed by Cloudflare"

"Instruct AI bot traffic with robots.txt" disabled

I think lying is a bit strong, I think they're potentially incorrect at worst.

The Cloudflare blog post where they announced this a few weeks ago stated "Cloudflare, Inc. (NYSE: NET), the leading connectivity cloud company, today announced it is now the first Internet infrastructure provider to block AI crawlers accessing content without permission or compensation, by default." [1]

I was also a bit confused by this wording and took it to mean Cloudflare was blocking AI traffic by default. What does it mean exactly?

Third party folks seemingly also interpreted it in the same way, eg The Verge reporting it with the title "Cloudflare will now block AI crawlers by default" [2]

I think what it actually means is that they'll offer new folks a default-enabled option to block ai traffic, so existing folks won't see any change. That aligns with text deeper in their blog post:

> Upon sign-up with Cloudflare, every new domain will now be asked if they want to allow AI crawlers, giving customers the choice upfront to explicitly allow or deny AI crawlers access. This significant shift means that every new domain starts with the default of control, and eliminates the need for webpage owners to manually configure their settings to opt out. Customers can easily check their settings and enable crawling at any time if they want their content to be freely accessed.

Not sure what this looks like in practice, or whether existing customers will be notified of the new option or something. But I also wouldn't fault someone for misinterpreting the headlines; they were a bit misleading.

[1]: https://www.cloudflare.com/en-ca/press-releases/2025/cloudfl...

[2]: https://www.theverge.com/news/695501/cloudflare-block-ai-cra...

> I think lying is a bit strong, I think they're potentially incorrect at worst.

I understand that you're trying to be generous, but the claim that "Matthew opted everyone in by default" is flat out incorrect.

We are in agreement; I just think saying "lying" implies a level malintent that isn't present -- it's rather overly ungenerous. The poster is at worst incorrect. And their misconception is understandable given the company's own confusing marketing.

I recently created a new Cloudflare account for a project I’m working on and moved two domains into it, and the settings were both on by default without asking me about it at all. The original press release specifically mentioned enabling it by default.

> Cloudflare, Inc. (NYSE: NET), the leading connectivity cloud company, today announced it is now the first Internet infrastructure provider to block AI crawlers accessing content without permission or compensation, *by default*.

> companies who want AI to recommend their products need to turn this off before it starts hurting them financially

Content marketing, gamified SEO, and obtrusive ads significantly hurt the quality of Google search. For all its flaws, LLMs don’t feel this gamified yet. It’s disappointing that this is probably where we’re headed. But I hope OpenAI and Anthropic realize that this drop in search result quality might be partly why Google’s losing traffic.

This has already started with people using special tags also people making content just for llms.

There is a standard for making content just for LLMs: https://llmstxt.org

From their example I don’t see any value in this on top of making and actually human friendly site.

> Converting complex HTML pages with navigation, ads, and JavaScript into LLM-friendly plain text is both difficult and imprecise.

None of these conditions should apply for websites with purpose of providing information.

I hope they realize Cloudflare opted them in to blocking LLMs.

I hope you realise that lying is bad.

Yeah, that's a lie. I didn't do anything and I didn't get opted in.

Edit: And, btw, that statement was true before the default was changed. So, your comment is doubly false.

Reply on news.ycombinator.com