Hacker News Reader

Comment by timpera | Hacker News Reader

Please don't use Anubis, it makes visiting websites very difficult (often multiple minutes wait times) on old hardware and low-end smartphones.

View on HN

As things stand right now, the current web may not be possible to maintain for old and low-end phones given the costs imposed by AI training crawlers.

Please propose an alternative to both Cloudflare and Anubis, that shields websites against inhuman traffic without frequent operator intervention (or otherwise negates the capacity costs they pay for AI crawling) and is compatible with low-end smartphones.

Certainly, I imagine Anubis would be interested in adopting it if it’s effective!

Great you now made the internet unusable for half the planet.

Sounds like something the government should regulate and own then rather than relying on corporate benefactors that all profit off of malware advertising.

That will be about as effective as attempts to block soccer streams in Spain are, and I don't think it'll pan out as a viable solution without worldwide enforcement backed by BGP bans of providers that continue to host non-compliant sites after notified. It's technically possible to consider a future where RIPE might withdraw your BGP assignment if you host advertising malware on your website, but ultimately that's just as dystopic as the DMCA is today, only scaled up to worldwide impacts and drama.

AI is absolutely going to make the internet unusable for half the planet if we don't find a better way; it's happening in realtime and I hate to see it. I would love to know of a better way, but everyone's so busy clamoring against effective solutions without proposing anything else that would work.

Doing nothing doesn't work. Doing Cloudflare isn't acceptable. Doing Anubis makes the Internet unusable "for half the planet". I'm on your side morally, but site operators can't afford to be idealists in the face of AI crawling bills.

What do you, does anyone, suggest that hasn't been tried, or adopted, or considered? What's left that will defend operators against AI's traffic flood that isn't eating a thousand doller crawler bill, paying Cloudflare, or locking out half the planet?

The solution is building simpler and more efficient websites. You won't win the arms race, so make it irrelevant. Any random VPS hosting a static website can already serve millions of requests an hour.

Personally I think that's a better outcome than the alternative.

However another option is to support both. Have a challenge page that requires the visitor to select one of several options. Cloudflare could be one of those.

What's the alternative? We're stuck, either cloudflare or anubis

what kind of hardware would take minutes to complete the checks?

It really depends on the difficulty level chosen by the website. I'm testing on both a Xiaomi Redmi 5 and my university library's computer.

Reply on news.ycombinator.com