Google's AI advantage: why crawler separation is the only path to fair Internet

The top ten from this PR piece, among other data points in there, seem to work against their argument imo. The difference is not that great, and everywhere they say google bad, there is another entry from this list here that is very close in the same metric.

What it sounds like they want is a version of the new permissioned / ethical licenses we have seen in OSS for website owners and their content. You made it public and certain things happen when you do that, another option is to require login or other gated access, which is the self-enforceable option

Personally, I want Google Ai accessing my content, but definitely not Perplexity, but also I'm not going to stop fair access to my content. I want people to learn from it by reading it directly or with / through an Ai. I don't care how they learn, everyone is different.

Cloudflare is not representing my interests in this. Recent support cases (still unresolved) show my how little CF cares about me as a paying user and this reinforces that sentiment and perspective (imo)

* unable to update my whois, their form is utterly broken, they keep sending me compliance email, email support was a prick, unresolved after 2 weeks now

I highly recommend removing CF from any considerations for (1) this (2) you will fight parity issues in runtimes and your language packages, it's not worth the pain (3) it's more expensive to run meaningful workloads, if you have light stuff it's still ok.

---

In rounded multiple terms, Googlebot sees:

vs. ~1.70x the amount of unique URLs seen by ClaudeBot;

vs. ~1.76x the amount of unique URLs seen by GPTBot;

vs. ~2.99x the amount of unique URLs by Meta-ExternalAgent;

vs. ~3.26x the amount of unique URLs seen by Bingbot;

vs. ~5.09x the amount of unique URLs seen by Amazonbot;

vs. ~14.87x the amount of unique URLs seen by Applebot;

vs. ~23.73x the amount of unique URLs seen by Bytespider;

vs. ~166.98x the amount of unique URLs seen by PerplexityBot;

vs. ~714.48x the amount of unique URLs seen by CCBot; and

vs: ~1801.97x the amount of unique URLs seen by archive.org_bot.

---

If anything, I am very surprised that Claude, OpenAI, Meta, and Microslop are so close in unique urls seen