I part of the reason is Capthas, something is needed but Capthas are a pain to use.
Keep in mind that for the average person (and even the average techy), they will not even see the captcha, because they have enough tracking them across the web that it will 'autoresolve' them as a human. I only see captcha's nowadays when I am working on web scraping automation or browsing from my VPS.
IP+subnet based throttling on the other hand is a perfectly working solution, optionally with a proof-of-work or login requirement for intensive POST requests only. Why is it a problem if only a few requests per day are made per home IP? Just how bad and inefficient is the web server?
Bots are not a second class client. They were here before AI, and they, like AI, work for humans. For the most part, they're not trolling the web autonomously.
[1] Proposed Amendment to the Telephone Consumer Protection Act of 1991 (2026-05-08) - https://www.karlbunch.com/random/website-protection-act/
Even a $5 VM I have has an included quota of at least 1TB egress data per month, and it can handle 9,540,534 requests in a day if not less. A Rust server can handle a lot more. In practice, my egress quota is larger by virtue of having a couple of more nodes.
If you actually cared, you would use something cheaper then AWS, ideally with zstandard compression, and a CPU efficient service that doesn't buckle under a moderate load.
> Bots are not a second class client. They were here before AI, and they, like AI, work for humans. For the most part, they're not trolling the web autonomously.
While I can agree with this in principle, due to vibeslopping in general there are a LOT more unruly scrapers browsing around, which tilts the scales quite a bit.