Cloudflare CEO is lying to you about the bot traffic jump
flyingpenguin.com
flyingpenguin.com
Update: I see the problem. Here's the full tweet: https://x.com/eastdakota/status/2062212701414187452
"Thought it would be end of 2027, then early 2027, but agentic traffic growing so fast that bots have now passed human traffic online for the first time in the Internet's history."
But the quoted segment in the article was just "…bots passed human traffic online for the first time in the Internet’s history."
It looks to me like the data supports "bots passed human traffic" but does NOT support "agentic traffic", since more of that traffic is from AI crawlers building indexes than from agents that are browsing the web on behalf of their owners.
If that's the point the article is trying to make then the headline is a little more supported, though I'd still say it's too hype-y a headline.
I guess a lot of this rests on what you assume "agentic traffic" to mean.
I think you are missing the fact that the dashboard has HTML pre-selected as a filter. Once you change that to all content types, you’ll see humans account for twice as much traffic as bots.
Note this part of the article:
> The CEO ignored the all-traffic number, on his own dashboard, and instead published the HTML-only number as a fact about the whole internet.
The most important thing is to link to your methodology, which Cloudflare's CEO did in this case.
It's easy to drastically underestimate the amount of bot traffic, because bots make efforts (of varying sophistication) to look human enough to evade blocking. That includes using fake user-agent strings corresponding to real browsers (often but not always with implausibly old version numbers), proxying through residential IPs, and sometimes using full headless browsers. In my own data, traffic from badly behaved browser-impersonation bots exceeds traffic from named scrapers like GPTBot by something like 10x.
The measured percentage of bot traffic is higher for HTML than for other content types because many bots will load an HTML page, and then not load the JS/CSS/image/etc resources it references. But these are the least-sophisticated and most-detectable bots.
57.141.0.42 - - [05/Jun/2026:19:50:19 +0000] "GET /mid/a017bc62-0982-42db-8403-241d69da8d0f@alexander-goetzenstein.my-fqdn.de HTTP/2.0" 303 0 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/craw...)"
57.141.0.48 - - [05/Jun/2026:19:50:22 +0000] "GET /group/comp.os.linux.advocacy/a/a236f5a5-63a4-4982-8bb6-07ffc684201b@googlegroups.com HTTP/2.0" 200 34838 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/craw...)"
57.141.0.55 - - [05/Jun/2026:19:50:23 +0000] "GET /group/alt.recovery.aa/a/ne6onq%24hpp%241@dont-email.me HTTP/2.0" 200 5606 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/craw...)"
57.141.0.56 - - [05/Jun/2026:19:50:24 +0000] "GET /group/aioe.news.assistenza/a/qpukie%241i1g%241@neodome.net?view=headers HTTP/2.0" 200 17027 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/craw...)"
57.141.0.36 - - [05/Jun/2026:19:50:29 +0000] "GET /group/alt.obituaries/a/uf8pej%241hqi1%241@news.xmission.com HTTP/2.0" 200 6123 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/craw...)"
57.141.0.66 - - [05/Jun/2026:19:50:29 +0000] "GET /group/comp.theory/a/v3640k%24vg63%243@dont-email.me HTTP/2.0" 200 148720 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/craw...)"
User-agent: meta-externalagent
Disallow: /
https://www.reddit.com/r/webdev/comments/1sdzd1q/metas_ai_cr...
User-agent: *
Disallow: /
User-agent: *
It's almost a full time job to manage. Our WAF rules are 90% 'oh shit AI scraper bots found a new vector.' It doesn't border on DDoS -- it effectively is. Coupled with all the Google changes that started a few years ago -- which is a separate topic I could rank about for ages -- 2026 is just a VERY bad time to be a website owner. I actually wiped most of our robots.txt rules the other day because literally nobody followed them. Anybody who tells you otherwise is flatly lying.
This is the new normal though, we gotta try and figure it out
If the "bot traffic" declines, then the "bot protection business" goes down with it
Cloudflare communication are sometimes careful to refer to traffic _labeled as_ bot traffic versus actual bot traffic
Because the "business" relies on the existance of "bot traffic", theres an incentive to broaden the scope of what is labeled as "bot traffic"
The false positive rate can be high. The public should see those statistics, and in truth it may be infeasible to compile them when theres no verification and the entire system relies on heuristics
"Bot protection" can be used to gather fingerprints for marketing
It can be used to force users to use certain software, e.g., certain browsers, and to enable Javascript subjecting users to data collection, surveillance and ads
Originally the motivation for avoiding "bot traffic" was based on behaviour, e.g., exceeding acceptable rates of usage, making too many requests in a given time period, exceeding rate limits
Now it's available to exclude traffic based on criteria such as what browser someone is using. NB. This is more than "user-agent string". The company forces people to sign NDAs before telling them what it is doing to fingerprint www users
If residential proxies are the problem then why not go after the companies that provide them
The truth is that those companies are not the problem. Their customers are so-called "tech" companies
Perhaps it's these so-called "tech" companies that are the problem
Certainly the problem is not the individual www user who doesnt use an "approved" graphical, Javascript-enabled browser who gets blocked or fingerprinted trying to make a single request
But thats who suffers from "bot protection" so that so-called "tech" companies can profit from data collection, surveillance and ads
> Now it's available to exclude traffic based on criteria such as what browser someone is using
I'm pretty sure user-agent-based bot detection predates every request-rate-based method by quite a few years.
>Certainly the problem is not the individual www user who doesnt use an "approved" graphical, Javascript-enabled browser who gets blocked or fingerprinted trying to make a single request
The alternatives to javascript fingerprinting are either ineffective (TLS fingerprinting and/or IP rate limits), or even worse for privacy (eg. attestation).
>If residential proxies are the problem then why not go after the companies that provide them
Javascript fingerprinting itself is ineffective, these kind of checks only stop the most basic bots and I'd argue the same for attestation.
Is data collection for "age verification" truly for age verification purposes only
Many bloggers and online commenters are not willing to accept that assumption
Is fingerprinting different
It is data collection for the purpose of establishing "online identity"
To the extent its based on user-agent strings or other HTTP headers it requires allowing a third party to MiTM TLS connections
Some bloggers and online commenters are opposed to computer users who MiTM _their own_ TLS connections
How is Cloudflare different
Mostly current valid user agents, lots of ip addresses, but the traffic patterns are not organic. I’m not clear if it’s bad ai scraping or dos, but at some level it’s indistinguishable.
The only thing I saw that could possibly be construed as abusive were some poorly configured RSS bots. Even when my server told the bot that the page would not change for 4 hours the RSS bots would check every 10 minutes meaning they are ignoring the cache-control header. This was entirely harmless, just slightly annoying. The RSS bots are not new. Most of the bots are not even trying to disguise themselves as humans. Most bots are not programmed to parse cache-controls, rel tags or fetch robots.txt meaning they only follow the pirate code. A bot will do what a bot can do.
I was expecting the bots to mirror a couple git repositories I exposed but they did not go deeper than the README.md. None of them. I think this is the same pattern of catastrophization that exists around AI dooming the world and I don't know why it is spreading. I guess it must work or people would not do it.
[1] - https://blawg.nochan.net/b/Internet-Crap/20260522-Maybe-AI-B...
I have also seen thousands of requests per hour from the IP to a small set of pages, e.g. the homepage. I don't know why; it doesn't matter so I ignore it.
I've recently found there are websites offering curated "AI ready" datasets, and several of these sites claim to have indexed our site, on the 3-4 I looked at it was one of a few hundred datasets. It's interesting enough to be something an AI company would want, so my conclusion is the site is being specifically targeted by the AI bot developers.
As I was reading your comment it sounded like a targeted attack. I think you are right that it was targeted. I assume you have done research on what content could be rate limited by URI target vs. source IP and give people a message saying content temporarily unavailable due to AI bot attack?
Is the concern that your site is being DDoS'd or that they are reselling your copyrighted material? If reselling I would get corporate lawyers involved and seek damages I am not a lawyer. Feds could subpoena some of the providers for identity of the attackers.
If the concern is DDoS have your team done any analysis of the clients to see what is in common? Based on the number of IP address you are talking about I assume it must be from wireless carriers. Have you looked at TCPSYN TTL and other characteristics? If there is anything in common those connections could be routed internally to another listener that has tighter rate limits meaning that perhaps cellular users could find some content not available until the bots go away or they randomly get one of a dozen different captchas or random javascript puzzles to access each document until the storm subsides. The puzzles could probably be regenerated hourly by AI to keep the attackers on their toes. Another option would be to require an account to access the documents and limit the number of documents each account can download per hour and / or day and / or week then add more friction to account creation or limit account creation to address space of countries you do business with after blocking most proxies and VPN providers.
Another option to limit the blast zone of an attack is to block countries that one does not do business in but that depends on your business model.
CDN's like Cloudflare are not doing anything magic. If they can block the bots so can just about anyone else. Without seeing samples of the attacks I could not make many more suggestions.
So far there's always been some pattern to allow a block/challenge, e.g. user agent, JA3 / JA4, ASN. (I haven't looked at TCP SYN TTL before.) Usually the IPs are 80% or so in one country (e.g. Brazil, US, Vietnam or India) with the rest all over the world, mostly consumer ISPs although I haven't distinguished between fixed line and mobile.
We tried Cloudflare for a couple of months, on a paid plan, which I think blocked many of the non-distributed crawlers, but didn't help much with these distributed ones.
Meanwhile we have been reducing the cost of rendering the pages.
It sounds like there is the option to at least reduce the load by up to 80%. That's at least a start.
This repo [1] is not perfect but it's a start. I would disable IPv6 access to the site after removing the IPv6 DNS records and waiting a day so that attackers are forced onto IPv4 and clone this repo [1] or use one of the GeoIP databases to limit access from specific countries.
That repo also contains known proxies. That may account for another percentage of that remaining 20%.
As the content is academic in nature I don't know how your team feels about blocking Tor, but there is also a list of many (not all) of the last 30 days of Tor exit nodes in that repo.
The blocks would not have to be permanent, just enabled during the storms if your team so desired.
# verify you have console access to the server before doing things like this:
for BadIP in $(grep -Ev ^# ipip_country_br.netset); do ip route add blackhole "${BadIP}" 2>/dev/null;done
As this is academia I don't know if there is a concept of service level agreement or a promise of availability, but during attacks the requests to specific URL's could be redirected to a static pre-compressed landing page served out of memory that says "Access to these documents limited during AI bot attacks, here is where to request a full download instead: "
I forgot to mention, many of the AI bots are limited to HTTP/1.1. During an attack your server could redirect those request to a static page as well. Curious if you can tell by your access logs if the majority of the attack is HTTP/1.1 or HTTP/2.0. Most botters are too lazy to implement HTTP/2.0. Real browser clients use HTTP/2.0. Example of using HTTP/1.1 and HTTP/2.0
# Fist use HTTP/2.0 with headers I expect.
curl -i --http2 \
-A "Mozilla/5.0 (X11; Linux x86_64; rv:151.0) Gecko/20100101 Firefox/151.0" \
-H "sec-fetch-mode: navigate" \
-H "accept-language: en-US,en;q=0.9" \
--url 'https://blawg.nochan.net/.env'
# Then HTTP/1.1 for the lulz, pay attention to the Location redirect
curl -i --http1.1 \
-A "Mozilla/5.0 (X11; Linux x86_64; rv:151.0) Gecko/20100101 Firefox/151.0" \
-H "sec-fetch-mode: navigate" \
-H "accept-language: en-US,en;q=0.9" \
--url 'https://blawg.nochan.net/.env'
[1] - https://github.com/firehol/blocklist-ipsets/
[2] - https://github.com/firehol/blocklist-ipsets/blob/master/ipip...
There is nothing on the site, just my name and an email account and it should have no traffic at all.
Instead in the last 30 days I've had 26K requests. 93% resulting in a 4xx error as most of these bots seem to be looking for vulnerabilities in various platforms like WordPress.
I would assume a lot of people running websites tend to think in pageviews, especially when dealing with bots because images and CSS files tend to be "cheap" static content but HTML requests are often dynamically generated.
It's also a single tweet that links to the data used to "disprove" it. Would be a weird way to lie.
It’d make sense as you might not want your bot to load everything a real human would do (ie: analytics, ads, unrelated files, etc..) and only focus on the content.
Also, am I the only one surprised that bot traffic is not the majority already? For my site, it’s x100 bots for every human.
[1] - https://blawg.nochan.net/b/Internet-Crap/20260606-How-To-Blo... [work in progress]
The fact is, Cloudflare is a man-in-the-middle. That's their focus, that's their purpose.
They will limit your local crawler from accessing pages. They will demand you use their crawler.
They will decrypt your traffic if they get a warrant. They always decrypt your traffic anyway, but they will give it to state actors if they demand it.
That's not to say anyone should break the laws, but the issue right now is that intellectual property is incompatible with what is coming with AI.
I don't hate on Cloudflare because it's a bad service. It's actually pretty good, but the fundamental problem is they make their purpose to be a single choke point of all data on the Web.
That's not right. It never was.
They don't see anything wrong with one entity controlling most of the internet traffic
Source? According to cloudflare, their crawling service don't get any special treatment from their WAF/CDN.
And what, I'm here to do research for your arguments?
Cloudflare provides services that any of their users may enable to charge crawlers for access. And they announced a while ago that this was going to be enabled by default on new domains, whether the customer asked for it or not (and there is plenty online about people wondering why they aren't being crawled). When it is enabled, Cloudflare either blocks, require registration of the crawler (forcing enabled billing), or provide one themselves: https://developers.cloudflare.com/changelog/post/2026-03-10-...
My compressed claim was close enough (in my opinion), given the substantive argument was broader: chokepoint, MITM, decryption, the IP vs AI tension.
It's the same pattern as Flock cameras quietly becoming the default license plate surveillance layer across US cities. Private infrastructure is becoming the substrate for something that used to be public or distributed, with the concentration happening faster than any deliberative process about whether we want it.
I'm willing to be wrong on any specific piece of this. But chasing down receipts for every line is effort, and I'm tired and old. The shape is what matters, and the shape is a man-in-the-middle.
I recall Facebook doing it years ago, I imagine they still do.
A 'pixel' is an unobtrusive (as in, not seen by the user like a banner ad is seen) asset* served on a web page that can cause the user's user agent to make an affirmative web request from you, a third party, so you know someone was at the site serving your pixel.
Typically used for:
- tracking in general, as well as more specifically:
- retargeting
- conversion
* Note: Doesn't have to be a literal pixel, but a literal transparent pixel is least likely to get blocked. Serve your pixels from the end of a parameterized path (/some/param/or/other/pixel.gif) and it's not seen as query string tracking either.
So many people have sketchy TV boxes or whatever other sketchy IOT decice that is a larp for using your network to sell bandwidth to proxy networks.
However, CF is unnecessarily making people on 5G connections from desktops do turnstiles as it looks like a scraper using a mobile proxy. This will become more and more of an issue as more laptops have 5G modems in them. Not sute how this WAF IP fingerprinting model survives widesprear CGNAT. I guess it will be an excuse to more intensly fingerprint us.
What makes you think this is the cause, rather than something more straightforward like: CGNAT means more users are sharing IPs, and there's a higher chance that the IP pool gets contaminated by bad behavior? Apparently cloudflare tries to detect CGNAT pools and give them more leeway, but at the same time they can't give them unlimited leeway.
[1] https://blog.cloudflare.com/detecting-cgn-to-reduce-collater...
Maybe from your IP block even. More common since you don't control that.
That is something that should not be allowed to exist. It's one of the reasons monopolies (or even majority-opolies) are bad. It's a weapon hanging on the wall, waiting to be used.
Are there network effects like what happens with Microsoft in the business computing space? With Microsoft, I'm also aware of a great amount of anti-competitive behavior, and though I haven't seen that from Cloudflare personally and haven't heard accusations of it, I also haven't paid attention.
When I learned econ 101 in high school there was a concept of a "natural monopoly" like an electricity utility, a concept that was probably mostly post-hoc rationalization of the regulatory structures that were chosen a century ago, but it at least was a coherent narrative. I can't see any coherent narrative about Cloudflare's services being a natural monopoly. So I'm left wondering if they are just way better at what they do than anybody else, and perhaps the space isn't big enough to drive a competitor to enter it?
I hope somebody on HN has a much better explanation of this than I do.
Everyone using the free service likes that, of course, but honestly I wish we'd make it illegal to do. It's heavily used as a way to steal small markets simply by being successful in a different large one.
I can see the benefit, but if you made free services illegal companies like google would just offer gmail for a penny.
You'd could go farther and say that no one should be allowed to provide a good or service for less than it costs them to provide it. That still has problems though. For example, Netflix sends servers to ISPs to set up on their network so that segments of that ISP's customers are pulling data from one of those boxes distributing the load. ISPs are happy to do it because it means fewer customers call them to complain about slow speeds/buffering issues with netflix. If netflix wanted to start selling CDN services to others, having all those boxes at every major ISP in the nation means the cost to deliver that service will be much lower for netflix than it would for some new start up. You can't really make it illegal to offer a service at a price lower than it would cost the poorest and least efficient/capable would-be competitor though.
We'd also risk losing a lot. Everyone would be forced to pay youtube some amount proportionate to what it costs for youtube to exist. There are also extremely expensive to run services like the internet archive which would be bad to price people out of
For Internet Archive: IA is a non-profit that currently runs fine on donations, not ads, and in a healthier environment they'd get subsidies because they serve a clear public benefit, one which the government also relies on fairly frequently. They're a wildly different category of business than Cloudflare, both obviously and legally, and I think they'd be fine. And we'd be significantly better off if they had competition, because that'd serve as a distributed, duplicated backup.
I'm not claiming it'd be trivial, or perfect. I'm claiming it'd be worth the effort.
I don't think advertising is inherently bad, and at some point you get ambiguous about "is this advertising or is it just describing something so I can find it in a search engine" and it's literally impossible to eliminate it all. and being able to tell people with problem X that solution Y exists can be a good thing, in moderation. it's more that the current maximally-invasive insanity and e.g. tons of highly distracting billboards are probably a net loss for humanity. São Paulo shows that it's quite nice without it, for example.
Which is very nearly the only reason to have governments: forcing restrictions that lead to better results in aggregate.
(malicious control is another reason, but you kinda can't argue public benefit there)
What regulation? Be specific.
CloudFlare provides significant utility to me. I chose to use them. Explain why you think someone else needs to butt into this relationship.
CF serves something it convinced customers they need.
Static blogs hiding behind bot protection (in some cases blocking legit users from GrapheneOS because it's difficult to fingerprint them) because someone convinced them they'll be DDoSed by bots otherwise is a loss to the Internet.
A lot of self-hosters running CF tunnels because they don't know better also contributes.
> It's more healthy to start the conversation of _why_ CF services are valuable.
Begging the question. It's what TFA is about - telling people they need CF.
A lot of self-hosters running CF tunnels because they don't know better also contributes.
Are you saying CF documentation is better than Computer Science / Networking education resources? Why don't people know better? I thought the tunnels are mostly used to bypass NAT's.
> Static blogs hiding behind bot protection
I'm not sure what is the proportion of the static vs dynamic sites, but I would argue that for wordpress CF is adding real value.
While not free, you can do with with TCP HAProxy streams on a cheap VPS. A lot of people using them to bypass NAT don't realise that Cloudflare decrypt the traffic on the way - that's what I meant about them not knowing better.
I’d blame the bad actors, rather than service providers that alleviate the problem.
I am unaware of such a capability of Cloudflare.
I believe it is the site administrators who have inserted Cloudflare in between their sites and their users.
Usually it is done for rational reasons of establishing a protection against bots. But what is less rational, in my opinion, is when everyone uses the same provider for that.
Because it indirectly turns Cloudflare into a monopoly. And monopolies often converge to a state when they start to abuse their position.
I assume there's some way to detect that mechanically, which is probably what you described. I just haven't yet seen anything that does this - hopefully it already exists, if not I should probably make it.
Actually I think this might not be too hard. Cloudflare has a relatively small, published list of IP addresses: https://www.cloudflare.com/ips/ and Firefox has a DNS API for extensions: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/Web... (and kinda mostly not chrome, dev channel only: https://developer.chrome.com/docs/extensions/reference/api/d... )
That's not too hard to plug together, and Firefox even has stuff like `["offline"]` so you can avoid sending any requests at all.
So yea, that strategy seems to work. PoC functional, gonna clean it up later.
If you want to be mad at somebody, then it should be at the AI companies for basically forcing the rest of the internet to use them. They have broken every non-enforceable contract that it was built upon. We went nearly 25 years without needing something a MTM solution like that, and I would say, in the last 5 or so, it has become a hard requirement
You get the numbers that Cloudflare tells you, but who knows if you can trust their stats after their CEO is apparently cherry-picking data to shape their product narrative?
I wouldn't trust a single thing coming out of his mouth.
I've been monitoring bot traffic on digital platforms for over 10 years. Sure, the crawler share is growing, some even with malicious intentions, and those I detect and block.
I disagree that this pain is worth the cost of making real people spend their life on verification.
Same general idea goes for any of the algorithmic driven platforms. The algorithms are ostensibly intended to surface organically discovered things by watching how people interact with things. That they are so susceptible to distortion through bot farms should be a lot more acknowledged than it is. People trust them far more than they should.
There is also a general cost of running things concern. It isn't like it is completely free to execute on bot traffic.
If the digital platform's storefront is their business, they could afford to spend some budget on bot detection. Bots still come from data center networks, sometimes render pages incompletely, request resources in bulk, and show enough patterns to be flagged internally.
If we look at a medium website, most random crawlers will come from Amazon, Microsoft, DigitalOcean, Hetzner, OVH, and a few other DC networks — these can be blocked easily without harming real users. The rest can be detected and cleaned up, even manually.
The math is simple: 20,000 visits a day at 15 seconds each = ~83 hours a day lost watching a Cloudflare logo, just because someone doesn't want to dig into the logs. I don't buy it.
There is also a bit of mixed incentives. Yes, it is the ad platform that is getting abused. But it is also the ad platform that is charging people based on abused practices.
And it isn't like this is completely made up. Just look at how facebook killed a lot of ton of people during the "pivot to video" programs. I don't know all of the details, as I was thankfully not in any of the involved industries, but my understanding is it is fairly well documented.
Edit: I changed an "isn't" to "is." I think I was trying to reword at one point, but left it in a way that is opposite what I meant.
I've been dealing with the web since the previous century and still haven't managed to build a website that could be hurt by scrapers visiting it.
If you went through the logs, you'd probably see that these bots are on a single IP or subnet, which can be easily detected and blocked instead of closing off search to non-registered users.
Our offending searches were coming from many addresses.
That's extremely far from all sites though. It's probably safe to say it's a severe minority, particularly when you ignore personal / non-profit-bringing sites. Tons of small and large sites run stuff like poorly-written wordpress or ruby on rails or thousands of microservices doing god knows what. A major increase in request volume on those can easily mean significant increases in hosting charges (e.g. small-% on big, many multiples on small) or significant effort in optimizing (which is expensive too).
Even on a very bad day, the server's page load time doesn't go over 1s.
However, it seems like I'm indeed looking at the problem through the wrong prism, as what I've seen from the comments suggests that the initial issue is performance, and the bots are what uncover it.
Optimization is very very much not a solved problem though, just look at basically all software ever written - it's written for an optimization priority and to a price point (whether commercial $$ or via personal time), and that target's value to its users has shifted rather dramatically.
I'm working on an open-source tool that could be useful for bot detection, but I'm still not confident that anyone would deploy it on-prem and make the setup/maintenance instead of just routing traffic through the cloud.
Perhaps performance as a KPI could work. Thanks!
As to if it'd get enough interest: yea no idea at all. I wish you luck tho! Clearly there's a need for this kind of thing.
1. https://play.tirreno.com (admin/tirreno)
Y'know what would be better? Making a site showcasing what tools are better than Cloudflare's services! And how to use them! And sharing said site so people know about them!