Tangential but all the pages I see protected by Anubis are just static HTML. But bot traffic is only harmful on dynamic pages right?
Also, while some pages may look like static HTML pages, they may be generated on the fly by an expensive and/or slow backend, which adds to the cost. I happen to maintain servers for academics and some content management systems are slow and have an expensive CPU cost. While it's OK for the low number of humans interested in the subjects they deal with, it's definitely not fit for massive bot scrapping. And before you ask, no, it's not always practical to have cache upfront or to pre-generate all pages.
The situation you are in is far less dire sounding when you consider that you have these options available to you.
Well, sounds like it's not your problem then.
When I see LKML using Anubis when the pages it serves are tens of kB, all presumably static, I wonder if they do it out of spite rather than to protect their servers.
As a user/reader/viewer I absolutely hate Anubis and usually turn around when I see it pop up (at least on my phone where it takes ages to compute), but with stats like that, I get why a site operator would resort to using it.
This drives me crazy with so-called SOTA LLMs that have "achieved AGI".
Fable, Sol, Astra, will start by trying to reverse engineer a binary to figure out how something works when software is open source and one search query away.
You let them know it's open source, and they will start using github API instead of just cloning and grepping.
> At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.
14 CPU cores total for running a website like kernel.org is laughable. This is not worth burning cycles in Anubis on client's devices, this is not worth the time of the engineer who worked on it. Provisioning more hardware would have been literally better for everyone.
Yes and that should be fixed before you subject real users to resource-wasting scripts.
And to answer your intended demand: this is in some cases impossible or unreasonable. And things were working fine before LLM DDoS.
It also depends on if you want your content to be scraped. Any bot ignoring robots.txt should be considered potentially malicious and worth considering such mechanisms for.
That's a trivially solved problem - have the web server refuse (or redirect if you're feeling nice) requests with unexpected query parameters. No reason to make things worse for real users before optimizing your site.
And even if there are actual parametric resources you want to serve, apply limits only to them and let users visit the rest without unnecessary bot checks.
I could probably also analyze all the paths across my servers and write some bespoke Nginx config files to filter out all requests with illegal queries, but that's harder, more fragile, and just leaves the servers open for the next time a scraper finds a new way to screw me over.
The bots I'm fighting have gone for the nuclear option, relying on botnets and spoofing to bypass basic filtering, so I don't see why I shouldn't do the same.
Also, if you are not using a big CDN (e.g. Cloudflare), you still need to pay for the traffic (even if you don't pay as much for the CPU). This is doubly true for smaller CDNs like Bunny.
I only hear bad things about bots lately, are there any benevolent bots?
I've done a tiny bit of scraping myself (usually to archive old websites before they die), and I've gotten banned a few times even though I wasn't hammering anything.
Looks like it's getting harder and harder to do scraping for legitimate purposes. Although I suppose legitimate is subjective :) Everything becomes valuable when it's old enough...