The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real...

Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed data"?

Unless you're reddit, X, Google, or Meta with scary unlimited budget legal teams, you have no power.

Great video: https://www.youtube.com/shorts/M0QyOp7zqcY

Privacy cannot exist in an environment where the host gets to decide who access the web page. I'm okay with rate limiting or otherwise blocking activity that creates too much of a load, but trying to prevent automated access is impossible withou preventing access from real people.

Cloudflare seems very vocal about its desire to become yet another digital gatekeeper as of late, and so is Google. I want both reduced to rubble if they persist in it.

(What are some good alternatives to Cloudflare?)

Another way the situation is similar: email delivery is often unreliable and hard to implement due to spam filters. A similar thing seems to be happening to the web.

An allowlist run by one company that site owners chose to engage with. But the irony of taking an ideological stance about fairness while using AI generated comics for blog posts…

The web doesn't need to know if you're a human, a bot, or a dog. It just needs to serve bytes to whoever asks, within reasonable resource constraints. That's it. That's the open web. You'll miss it when it's gone.

>We need protocols, not gatekeepers.

But until we have working protocols, many webmasters literally do need a gatekeeper if they want to realistically keep their site safe and online.

I wish this weren't the case, but I believe the "protocol" era of the web was basically ended when proprietary web 2.0 platforms emerged that explicitly locked users in with non-open protocols. Facebook doesn't want you to use Messenger in an open client next to AIM, MSN, and IRC. And the bad guys won.

But like I said, I hope I'm wrong.

At some point soon, if not now, assume everything is generated by AI unless proven otherwise using a decentralized ID.

Likewise, on the server side, assume it’s a bot unless proven otherwise using a decentralized ID.

We can still have anonymity using decentralized IDs. An identity can be an anonymous identity, it’s not all (verified by some central official party) or nothing.

It comes down to different levels of trust.

Decoupling identity and trust is the next step.

Perhaps a way to serve ads through the agents would be good enough. I'd prefer that to be some open protocol than controlled by a company.

Most interaction, publication, and dissemination takes place behind authentication:

Most social media, newspapers, etc. throttle, block, or otherwise truncate non-authenticated clients.

Blogs are an extremely small tranche of information that the average netizen consumes.

I’ve actually just built an “identity token” using biscuit that you can delegate however you want after. So I can authenticate (to my service, but it could be federated or something just as well), get a token, then choose to create a delegated identity token from that for my agent. Then my agent could do the same for subagents.

In my system, you then have to exchange your identity token for an authorization token to do anything (single scope, single use).

For the internet, I’ve wondered about exchanging the identity token + a small payment (like a minuscule crypto amount) for an authorization token. Human users would barely spend anything. Bots crawling the web would spend a lot.

Just because you can, doesn't mean you should and I don't feel any one entity (private or public) should be an arbiter on these matters.

This is something that can, and should, be negotiated at the "last virtual mile".

and isn't this why people sign up with Cloudflare in the first place? for bot protection? to me, this is just the same, but with agents.

i love the idea of an open internet, but this requires all party to be honest. a company like Perplexity that fakes their user-agent to get around blocks disrespects that idea.

my attitude towards agents is positive. if a user used an LLM to access my websites and web apps, i'm all for it. but the LLM providers must disclose who they are - that they are OpenAI, Google, Meta, or the snake oil company Perplexity

I don't agree that bot identity is a problem or something that we are better off with if it is solved. I'd rather have a web where adversarial interoperability is possible than one where service operators have a say what tools you can use to access their websites.

The main problem with we are see today is bad actors that

a) completely ignore and/or side step copyright an licensing to use work of others for their own benefit without contributing anything back

b) send an unreasonable number of request

Both will not be solved with identifying bots, that will at best get rid of the small players and give Google, Meta, etc. even more power. The unsustainable parasitic theft of open content is something that needs to be dealt with legally and nothing else will solve it. DRM never works. If enough people block Gemini, Google will just feed it with Google bot crawls and no one can afford to block that. And then they will sell the data to other players. Or someone will make a browser extension to do the same.

The second issue also should be solved via legislation and enforcement thereof. It can also be solved by disconnecting and/or throttling abusive networks wholesale - whole countries if need be. You know, like we have been handling abusive network participants forever. Trying to detect "bots" is a fools errand that will only get rid of the laziest crawlers. You cannot win the bot blocking game when the bots can afford to spend more resources per request than real users are willing to.

So yes, the web must remain open. But to do that we must not have bot identity checks at all - whether that's managed by a single company that has inserted it as a gatekeeper or not.

I kind of want my site to be indexed with agents and used without any interference

You can use the admin certificate issued to you, to issue a certificate to the agent which will contain an extension limiting what it can be used for (and might also expire in a few hours, and also might be revoked later). This certificate can be used to issue an even more restricted certificate to sub-agents.

This is already possible (and would be better than the "fine-grained personal access tokens" that GitHub uses), but does not seem to be commonly implemented. It also improves security in other ways.

So, it can be done in such a way that Cloudflare does not need to issue authorization to you, or necessarily to be involved at all. Google does not need to be involved either.

However, that is only for things where would should normally require authorization to do anyways. Reading public data is not something that should requires authorization to do; the problem with this is excessive scraping (there seems to be too many LLM scraping and others which is too excessive) and excessive blocking (e.g. someone using a different web browser, or curl to download one file, or even someone using a common browser and configuration but something strange unexpected happens, etc); the above is something unrelated to that, so certificates and stuff like that does not help, because it solves a different problem.

They were working on an idea that looked a bit like an RSS feed for an entire website, where you would run your own spider and then our search engine could hit an endpoint to get a delta instead of having to scan your entire site.

If they’d made the protocol open instead of proprietary, we maybe could have gotten spiders to play nicer since each spider after the first would be cheaper, and eventually maybe someone could build pub sub hooks into common web frameworks to potentially skip the scan entirely for read-mostly websites, generating delta data when your data changed.

But of course when the next round of funding came due nobody was buying.

I thought about this a lot on my last project, where spiders were our customers’ biggest users. One of those apps where customer interactions were intense but brief and the rank in Google mattered equally with all other concerns. Nobody had architected for the actual read/write workflow of the system of course, and that company sold to a competitor after I left. Who migrated all customers to their solution and EOLed ours for being too fat in a down economy.

One of the hard parts in this space is what level of transparency should you have. We're advancing the thesis that behavioral biometrics offers robust continuous authentication that helps with bot/human and good/bad, but people are obviously skeptical to trust black-box models for accuracy and/or privacy reasons.

We've defaulted to a lot of transparency in terms of publishing research online (and hopefully in scientific journals), but we've seen the downside: competitors fake claims about their own best in-house behavioral tools that is behind their company walls in addition to investors constantly worried about an arms race.

As someone genuinely interested (and incentivized!) to build a great solution in this space, what are good protocols/examples to follow?

Joking aside, I think the ideas and substance are great and sorely needed. However, I can only see the idea of a sort of token chain verification as running into the same UX problems that plagued (plagues?) PGP and more encryption-focused processes. The workflow is too opaque, requires too much specialized knowledge that is out of reach for most people. It would have to be wrapped up into something stupid simple like an iOS FaceID modal to have any hope of succeeding with the general public. I think that's the idea, that these agents would be working on behalf of their owners on their own devices, so it has to be absolutely seamless.

Otherwise, rock on.

https://anchorbrowser.io/blog/page-load-reliability-on-the-t...

Here's to working together to develop a new protocol that works for agents and website owners alike.

bots vs humans:

humans are trying to buy tickets that were sold out to a bot

data scrapping:

you index my data (real estate listing) to not to route traffic to my site as people search for my product, as a search engine will do, rather to become my competitor.

spam (and scam): digital pollution, or even worse, trying to input credit card, gift cards, passwords, etc.

(obviously there are more, most which will fall into those categories, but those are the main ones)

now, in the human assisted AI, the first issue is no longer an issue, since it is obvious that each of us, the internet users, will soon have an agent built into our browser. so we will all have the speedy automated select, click and checkout at our disposal.

Prior to LLM era, there were search engines and academic research on the right side of the internet bots, and scrappers and north to that, on the wrong side of the map. but now we have legitimate human users extending their interaction with an LLM agent, and on top of it, we have new AI companies, larger and smaller which thrive for data in order to train their models.

Cloudflare simply trying to make sense of this, whilst maintaining their bot protection relevant.

I do not appreciate the post content whatsoever, since it lacks or consistency and maturity (a true understanding of how the internet works, rather than a naive one).

when you talk about "the internet", what exactly are you referring to? a blog? a bank account management app? a retail website? social media?

those are all part of the internet and each is a complete different type of operation.

EDIT:

I've written a few words about this back in January [1] and in fact suggested something similar:

    Leading CDNs, CAPTCHA providers, and AI vendors—think
    Cloudflare, Google reCAPTCHA, OpenAI, or Anthropic
    could collaborate to develop something akin to a
    “tokenized machine ID.”

how does this person think jwt’s work?

If this was cloudflare going into some centralized routing of the internet and saying everything must do X then that would be a lot more alarming but at the end of the day the internet is decentralized and site owners are the ones who are using this capability.

Additionally I don't think that I as an individual website owner would actually want / be capable of knowing which agents are good and bad and cloudflare doing this would be helpful to me as a site owner as long as they act in good faith. And the moment they stop acting in good faith I would be able to disable them. This is definitely a problem right now as unrestricted access to the bots means bad bots are taking up many cycles raising costs and taking away resources from real users

Web Bot Auth

https://news.ycombinator.com/item?id=45055452

and associated blog post:

The age of agents: cryptographically recognizing agent traffic

https://blog.cloudflare.com/signed-agents/

Basically MS tried to kill the web with their Win95 release, the infamous Internet Explorer and their shitty IIS/Frontpage tandem.

I deeply hate them since that day.

one potential solution: https://www.l402.org/

That is a very small part of the world we're entering.

The other vast majority of use cases will come from even more abusive bots than we have today, filling the internet with spam, disinformation, and garbage. The dead internet is no longer a theory, and the future we're building will make the internet for bots, by bots. Humans will retreat into niche corners of it, and those who wish to participate in the broader internet will either have to live with this, or abide by new government regulations that invade their privacy and undermine their security.

So, yes, confirming human identity is the only path forward if we want to make the internet usable by humans, but I do agree that the ideal solution will not come from a single company, or a single government, for that matter. It will be a bumpy ride until we figure this out.

It’s somewhat ironic to let fly the “free and open internet” battle cry on behalf of an industry that is openly destroying it.

Well, that's true of any crytpographic key?

In this case, it would mean you are giving them permission to act on your behalf. Nothing wrong with that.

If some of the people acting on your behalf start acting maliciously, then presumably those who decided to trust the people who were acting on your behalf would stop doing so.

Is this not common to how most any digital authentication works at all? You can always share your keys. That's a feature not a bug, when the actor you want to identify is meant to have a distributed implementation.

I understand the concern about how much power CloudFlare has, how they have the ability to gatekeep a large part of the internet. Absolutely, this is alarming.

But the Web Both Auth protocol itself is not the problem -- it seems to me to be written and designed appropriately for authentication of automated web agents.

And I think we desperately need something for that. I, like many people, are being forced to put bot precautions in place, because otherwise my sites are overwhelmed. But this means I wind up blocking bots that I don't want to block too. Because they are are partners, because I approve of what they are doing, becuase they have demonstrated good behavior. I have no way to do that right now.

IP address ranges are absolutely not the right way. IP addresses are network topology, not authentication. i worked in academia for some time, where large unviersities have a history of trying to use IP addresses for authentication -- and even working with internal IP addresses theoretically controlled by the (large) institution, it was a fool's game. IP addresses can change all the time -- even for a device which has not moved it's physical location. Plus resources can be allocated to different physical locations. Different actors can share an IP address. They are often changed at various lower levels of hiearchical administration without informing the top, for network topological concerns -- they are designed for this. Etc etc etc.

I understand the concern about CloudFlare's gatekeeping monopoly.

There may be ways that Web Both Auth can make it worse. Discussion of that is not inappropriate. Maybe there are ways to ameliorate it (will individual customers be ablet o have their own allow-lists? Can we insist on that? Is that enough?). Maybe not good enough. But let's focus the discussion on that -- there is in fact nothing wrong with Web Both Auth protocol, at least nothing covered in this essay, it is well-designed for authenticating bot agents, and we actually do need something that does that, in the current world where misbehaving disguised bot agents have become a real problem.

Not having a way to authenticate distributed bot actors who wish to opt in to a way to be authenticated (everyone else is free to try to evade the bot detectors same as they are now?) -- is going to create more damage. All these people railing against what seems to be an appropriate protocol for authentication because they don't like Cloudflare's monopoly are distressing me, it's going to be worse if we don't have a way to do it. It is an open protocol not just for use by cloudflare.

This isn't the problem Cloudflare are trying to solve here. AI scraping bots are a trigger for them to discuss this, but this is actually just one instance of a much larger problem — one that Cloudflare have been trying to solve for a while now, and which ~all other cloud providers have been ignoring.

My company runs a public data API. For QoS, we need to do things like blocking / rate-limiting traffic on a per-customer basis.

This is usually easy enough — people send an API key with their request, and we can block or rate-limit on those.

But some malicious (or misconfigured) systems, may sometimes just start blasting requests at our API without including an API key.

We usually just want to block these systems "at the edge" — there's no point to even letting those requests hit our infra. But to do that, without affecting any of our legitimate users, we need to have some key by which to recognize these systems, and differentiate them from legitimate traffic.

In the case where they're not sending an API key, that distinguishing key is normally the request's IP address / IP range / ASN.

The problematic exception, then, is Workers/Lambda-type systems (a.k.a. Function-as-a-Service [FaaS] providers) — where all workloads of all users of these systems come from the same pool of shared IP addresses.

---

And, to interrupt myself for a moment, in case the analogy isn't clear: centralized LLM-service web-browsing/tool-use backends, and centralized "agent" orchestrators, are both effectively just FaaS systems, in terms of how the web/MCP requests they originate, relate to their direct inbound customers and/or registered "agent" workloads.

Every problem of bucketing traditional FaaS outbound traffic, also applies to FaaSes where the "function" in question happens to be an LLM inference process.

"Agents" have made this concern more urgent/salient to increasingly-smaller parts of the ecosystem, who weren't previously considering themselves to be "data API providers." But you can actually forget about AI, and focus on just solving the problem for the more-general category of FaaS hosts — and any solution you come up with, will also be a solution applicable to the "agent formulation" of the problem.

---

Back to the problem itself:

The naive approach would be to block the entire FaaS's IP range the first time we see an attack coming from it. (And maybe some API providers can get away with that.)

But as long as we have at least one legitimate customer whose infrastructure has been designed around legitimate use of that FaaS to send requests to us, then we can't just block that entire FaaS's IP range.

(And sure, we could block these IP ranges by default, and then try to get such FaaS-using customers to send some additional distinguishing header in their requests to us, that would take priority over the FaaS-IP-range block... but getting a client engineer to implement an implementation-level change to their stack, by describing the needed change in a support ticket as a resolution to their problem, is often an extreme uphill battle. Better to find a way around needing to do it.)

So we really want/need some non-customer-controlled request metadata to match on, to block these bad FaaS workloads. Ideally, metadata that comes from the FaaS itself.

As it turns out, CF Workers itself already provides such a signal. Each outbound subrequest from a Worker gets forcibly annotated "on the way out" with a request header naming the Worker it came from. We can block on / rate-limit by this header. Works great!

But other FaaS providers do not provide anything similar. For example, it's currently impossible to determine which AWS Lambda customer is making requests to our API, unless that customer specifically deigns to attach some identifying info to their requests. (I actually reported this as a security bug to the Lambda team, over three years ago now.)

---

So, the point of an infrastructure-level-enforced public-visible workload-identity system, like what CF is proposing for their "signed agents", isn't just about being able to whitelist "good bots."

It's also about having some differentiable key that can cleanly bucket bot traffic, where any given bucket then contains purely legitimate or purely malicious/misbehaving bot traffic; so that if you set up rate-limiting, greylisting, or heuristic blocking by this distinguishing key, then the heuristic you use will ensure that your legitimate (bot) users never get punished, while your misbehaving/malicious (bot) users automatically trip the heuristic. Which means you never need to actually hunt through logs and manually blacklist specific malicious/misbehaving (bot) users.

If you look at this proposal as an extension/enhancement of what CF has already been doing for years with Workers subrequest originating-identity annotation, the additional thing that the "signed agents" would give the ecosystem on behalf of an adopting FaaS, is an assurance that random other bots not running on one of these FaaS platforms, can't masquerade as your bot (in order to take advantage of your preferential rate-limiting tier; or round-robin your and many others' identities to avoid such rate-limiting; or even to DoS-attack you by flooding requests that end up attributed to you.) Which is nice, certainly. It means that you don't have to first check that the traffic you're looking at originated from one of the trustworthy FaaS providers, before checking / trusting the workload-identity request header as a distinguishing key.

But in the end, that's a minor gain, compared to just having any standard at all — that other FaaSes would sign on to support — that would require them to emit a workload-identity header on outbound requests. The rest can be handled just by consuming+parsing the published IP-ranges JSON files from FaaS providers (something our API backend already does for CF in particular.)

The real question is whether there is more business opportunity in supporting "unsigned" agents than signed ones. My hope is that the industry rejects this because there's more money to be made in catering to agents than blocking them. This move is mostly to create a moat for legacy business.

Also, if agents do become the de-facto way of browsing the internet, I'm not a fan of more ways of being tracked for ads and more ways for censorship groups to have leverage.

But the author is making a strawman argument over a "steelman" argument against signed agents. The strongest argument I can see is not that we don't need gatekeepers, but that regulation is anti-business.

I'm sorry, but the "agents" of "agentic AI" is completely different from the original purpose of the World-Wide Web which was to support user agents. User agents are used directly by users—aka browsers. API access came later, but even then it was often directed by user activity…and otherwise quite normally rate-limited or paywalled.

The idea that now every web server must comply with servicing an insane number of automated bots doing god-knows-what without users even understanding what's happening a lot of the time, or without the consent of content owners to have all their IP scraped into massive training datasets is, well, asinine.

That's not the web we built, that's not the web we signed up for; and yes, we will take drastic measures to block your ass.