Hacker News Reader

We have Web Bot Auth to allow good bots to identify themselves to website operators: https://developers.cloudflare.com/bots/reference/bot-verific...

Bot detection is a big problem to solve, but it’s a significant focus at Cloudflare. (It’s not my team at Cloudflare specifically, but we work closely with them)

Options:

1. Any cost of browsing an e-commerce site is taken off the next purchase, whenever it happens.

2. Give each user 100 free page viewed per day or some such before you charge.

3. You don’t actually have to charge users for browsing the site if you provide a free or cheap API allowing bots to search and index your entire catalog. Agents and bots would certainly rather parse a kilobyte of JSON than 20 megabytes of HTML generated by on page JavaScript.

4. If you don’t like this system you don’t have to participate. If Amazon wants to do their own thing, they can. But if you publish a blog and want to charge $0.00001 per page view and browsers support this out of the box, why not?

1. Bots don't make purchases (and you can't identify them anyway), so there's nothing to take the costs off of.

2. Again, no way to identify users (bots use hordes of residential proxies with only a few requests per IP).

3. Agents and bots care more about a universal solution that works for all the sites than an efficient solution. You could standardize on a header for this, but then some sites would start hiding some content in the API for "business reasons", making this header untrustworthy. Unless you're a site so large that it deserves special handling, it ain't gonna work.

>1. Bots don't make purchases (and you can't identify them anyway), so there's nothing to take the costs off of.

yes they do. Claude Code with Opus 4.6 bought a U.S. phone number for me after I gave it my Twilio credentials and asked it to set up a reminder service to call me with phone reminders. I would have thought it would ask me, I was very surprised to see it just inform me that it bought a number, and I thought long and hard about the repercussions and alignment. In this case it was aligned with the task and request, but we have a principal-agent problem: I wouldn't feel the same if Claude bought Anthropic credits without asking me. ("Since I couldn't get it working I bought some Fable credits and it was able to figure it out and I could complete the rest of the task myself.")

Regarding 1, a significant portion of what I’d like my agents doing for me is helping me make purchasing decisions. It’s a lot of what I’m sending them out to research. So anything that’s making that harder is disadvantaging itself in those purchasing decisions.

Wouldn’t (1) be the point?

That's just going to be a really different from the shopping experience customer's are used to today, and I don't think customers would go for it. I know for at company asking customers to pay for the shopping experience would be a non-starter... If bot traffic became untenable we would probably do something like required account creation + sms verification, and even that would be a huge change in expectations for our customers.

Well it sounds like it’s either that or tragedy of the commons where Amazon et al needs to charge higher and higher prices due to inefficient bots constantly crawling their site. Also consumers are already about to change how they shop. In 3 years nobody will be firing up the browser to go to Amazon.com. They will be asking an agent to “buy the cheapest 3 ply toilet paper from a brand name and some decent flossers. Spend no more than $30 total.” For this kind of shopping the cost of the search is built into the purchase price one way or another.

You just described my own personal vision of hell

lol, no!

Having written bots several times, any kind of friction or payment on the json api would make me just use the free html "API" it's just easier.

I have many times used a webpage as api instead of the actual api because using the actual api required doing paperwork, like writing business cases, filling out approval forms, creating accounts, paying, etc...

I would like to argue that trying to provide a free service is non achievable, most of the time it will drill down to ads, people are already paying electricity and time in ads. If we pay say 3 secs of compute time of monero, and everyone pay the same... you remove the ads from the internet, people will start gettind paid without gate keepers for content they generate, and you can charge the AI machine for ingesting your content.

We were providing free services decades ago. Hosting a website, or a Minecraft server, or a VOIP server, or IRC, or a forum simply doesn't cost that much. Well within "some guy's hobby budget" type expenses.

> We were providing free services decades ago.

That was then.

Now VPS providers are significantly increasing prices (Due to memory shortage) making it unaffordable to run servers cheaply.

Servers, and memory, are orders of magnitude cheaper today than they were decades ago... for memory see: https://dam.stanford.edu/memory-prices.html

Yes there's a slight upwards blip right now, but not even close to cancelling decades of progress in price reduction.

Yet absolute price per GB isn't the full story when the popular tools accessible for the general public require more RAM for the same task today as compared to the past. You won't get far using the latest version of MS Word on a desktop with specs from 1999, or running MS Word 97 to edit academic essays turned this year.

Even for background services, rather than the general public, I would not attempt running an email server with the same RAM as I used in the 90s, 00s, or 10s.

The tools for hosting a website have barely gone up in requirements.

And for those documents I bet you could convert them and they'd work fine.

Yeah, it went from ~$10 / month to ~$30 a month to rent a relatively beefy VPS capable of doing the above (game servers being the most resource demanding of the lot). Still well within hobby money.

They're not more expensive than a decade ago, when we were still using them.

DO Droplets are still $5/mo, as are OVH VPS’es.

> And if we can't tell bot from human, why would bots choose to pay rather than just use the public endpoints we serve to our customers?

Assuming technical indistinguishability, the only solution is what was originally proposed for email: balanced net $0 charges for "normal user" usage patterns (i.e. payments from - payments to = $0).

If you x402 everything, and an average user access 5 pages, but a bot accesses 500 (or 5x100 times), then you've still achieved a substantial price delta that you could offset via a rebate

The real rub is about uniqueness attribution, as being able to differentiate 20 distinct real users from 1 bot w/ 20 proxies is the crux of anything above.

Asking customers to pay anything prior to purchase is a non-starter... Imagine a brick-and-mortar shoe store asking customers to pay an entry fee to shop before customers enter the door?

Cover charge for a shoe store is an interesting idea. I hate shopping, especially with crowds, and probably wouldn't mind, especially if it went towards a purchase, even partially. (This willingness is learned, probably only true for me since my 30s, and not strictly correlated to my income bracket.)

I often buy gas at the BP with a $0.03/gal higher price than the Shell across the street. Guess which one has a longer line more full of gas guzzlers.

A commercial site being invisible to AI Agents is starting to be as smart as being invisible to search engines.

SEO is standard practice, whatever the same thing is for Agents will be as standard very soon.

More and more people go "I want to buy thing X" to an AI Agent and if it can't find or access your product pages, it practically doesn't exist for those people.

The problem is that AI has made it increasingly difficult to tell bot from human.

Presumably Cloudflare's answer to this is CAPTCHAs.

Some of the agents out there today can bypass Cloudflare's CAPTCHA challenges.

You forget that bots are often just AI agent working for humans. This will be more and more common. So by blocking bots (which is literally impossible but lets assume otherwise) you are also blocking legitimate customers.

This can be solved by implementing RfC 3514.

What if they give in and just expose their database directly?

This is basically the same problem as bear-safe trash cans - there's substantial overlap between the smartest bears and stupidest humans. Affordances that one audience can use and the other can't (requiring human finger dexterity) are the only real solution.