Comment by jt2190 | Hacker News Reader

Comment by jt2190 | Hacker News Reader

Parent Full thread

The “client-side problems” Siebenmann is talking about are the various anti-bot measures (CAPTCHAs, rate limiters, etc.) that operators put in place that make the end user experience worse. Operators feel that they have no choice but to keep their servers available, thus they “don’t care”.

He makes a statement in an earlier article that I think sums things up nicely:

> One thing I've wound up feeling from all this is that the current web is surprisingly fragile. A significant amount of the web seems to have been held up by implicit understandings and bargains, not by technology. When LLM crawlers showed up and decided to ignore the social things that had kept those parts of the web going, things started coming down all over the place.

This social contract is, to me, built around the idea that a human will direct the operation of a computer in real time (largely by using a web browser and clicking links) but I think that this approach is extremely inefficient of both the computer’s and the human’s resources (cpu and time, respectively). The promise of technology should not be to put people behind desks staring at a screen all day, so this evolution toward automation must continue.

I do wonder what the new social contract will be: Perhaps access to the majority of servers will be gated by micropayments, but what will the “deal” be for those who don’t want to collect payments? How will they prevent abuse while keeping access free?

[1] “The current (2025) crawler plague and the fragility of the web”https://utcc.utoronto.ca/~cks/space/blog/web/WebIsKindOfFrag...

> The “client-side problems” Siebenmann is talking about are the various anti-bot measures (CAPTCHAs, rate limiters, etc.)

Directly from the article:

> it's not new, and it goes well beyond anti-crawler and anti-robot defenses. As covered by people like Alex Russell, it's routine for websites to ignore most real world client side concerns (also, and including on desktops). Just recently (as of August 2025), Github put out a major update that many people are finding immensely slow even on developer desktops.

The things he links to are about things that are unrelated to anti-bot measures.

The fact is, the web is an increasingly unpleasant place to visit. Users are subject to terrible UX – dark patterns, tracking, consent popups, ads everywhere, etc.

Then along come chatbots and when somebody asks about something, they are given the response on the spot without having to battle their way through all that crap to get what they want.

Of course users are going to flock to chatbots. If a site owner is worried they are losing traffic to chatbots, perhaps they should take a long, hard look at what kind of user experience they are serving up to people.

This is like streaming media all over again. Would you rather buy a legit DVD and wait for it to arrive in the post, then wait through an unskippable lecture about piracy, then wait through unstoppable trailers, then find your way through a weird, horrible DVD menu… or would you rather download it and avoid all that? The thing that alleviated piracy was not locking things down even more, it was making the legitimate route more convenient.

We need to make websites pleasant experiences again, and we can’t do that when we care about everything else more than the user experience.

The chat bot operator slurps all websites and gives answers to all questions free of charge.

No other website can compete with that.

The whole story with streaming media is not just that pay streaming became more convenient. It’s also that content creators used legal and business mechanisms to make piracy inconvenient. They shut down Napster. They send DMCA notices. They got the DMCA enacted. They got YouTube working for them by serving ads with their content and thus monetizing it.

Chat bots are just like Napster. They’re free-riding off the content others worked to create. Just like with Napster, making websites more convenient will be only part of the answer.

> content creators used legal and business mechanisms to make piracy inconvenient

Copyright holders, not content creators. Though typically content creators are also copyright holders, copyright holders are not always content creators, esp in this context. To a big degree these practices are not on the behalf of content creators nor are they helping them.

The solution may be elsewhere: starting from creating content that people may actually care about.

> The chat bot operator slurps all websites and gives answers to all questions free of charge.

> No other website can compete with that.

Copyright infringers uploaded music, television, and films free of charge, yet people still pay for all of that.

> The whole story with streaming media is not just that pay streaming became more convenient. It’s also that content creators used legal and business mechanisms to make piracy inconvenient.

Do you seriously think that copyright infringement ended when Napster went down? Have you never heard of the Pirate Bay or Bittorrent? They didn’t succeed at all in shutting down copyright infringement. People pay for things because it’s convenient, not because copyright infringement is no longer an option.

Another implicit social contract is that you can tell whether a request is coming from a commercial or non-commercial source based on the originating ISP. This was always a heuristic but it was more reliable in the past.

If 1000 AWS boxes start hammering your API you might raise an eyebrow, but 1000 requests coming from residential ISPs around the world could be an organic surge in demand for your service.

Residential proxy services break this - which has been happening on some level for a long time, but the AI-training-set arms race has driven up demand and thus also supply.

It's quite easy to block all of AWS, for example, but it's less easy to figure out which residential IPs are part of a commercially-operated botnet.

> it's less easy to figure out which residential IPs are part of a commercially-operated botnet

Is the client navigating the site faster than humanly possible? It's a bot. This seems like a simple test.

> 1000 requests coming from residential ISPs around the world could be an organic surge

But probably isn't.

> This seems like a simple test.

Not when the singular bot has a pool of millions of IPs to originate each request from.

If you think there's an easy solution here, productize it and make billions.

IPs that you've never seen before are hitting a single random page deep within your site are bots, or first-time followers of a search engine link. Grey list them and respond slowly. If they are seen again at normal human rates, unthrottle them.

Aka, make the web worse for everyone. Thanks for playing.

The Internet and Web were both designed with the assumption of cooperation. I wonder what they would have built if they'd taken hostility into account from day one.

As we've seen security is really hard to build in after the fact. It has to be part of your design concept from the very first, and pervades every other decision you make. If you try to layer security on top you will lose.

Of course you may discover that a genuinely secure system is also unusably inconvenient and you lose to someone willing to take risks, and it's all moot.

> > One thing I've wound up feeling from all this is that the current web is surprisingly fragile. A significant amount of the web seems to have been held up by implicit understandings and bargains, not by technology.

This is something I've been pondering, and honestly I feel like the author doesn't go far enough. I would go as far as to say a lot of our modern society has been held up by these implicit social contracts. But nowadays we see things like gerrymandering in the US, or overusing the 49-3 in France to pass laws despite the parliament voting against them. Just an overall trend of only feeling constrained by the exact letter of the law and ignoring the spirit of it.

Except it turns out these implicit understandings that you shouldn't do that existed because breaking them makes life shittier for everyone, and that's what we're experiencing now.

The only possible contract moving forward will be something along the lines of agreeing to paying for access. I can't imagine how this will work, but with the collapse of the ad-supported Internet, that's the only way forward. It's not a straightforward problem to solve because people are not willing to pay for something before they see it, but after seeing the page, the incentive to pay is gone.

Reply on news.ycombinator.com