You basically need proof-of-work to make this work. Idling a connection is not computationally expensive, so is not a deterrent.
It's a shitty solution to an even shittier reality.
It's a shitty solution to an even shittier reality.
Basically what they said. This is a hack, and it's specifically designed to exploit the infrastructure behind industrial-scale scraping. They usually have a different IP address do the scraping for each page load _but share the cookies between them_. This means that if they use headless chrome, they have to do the proof of work check every time, which scales poorly with the rates I know the headless chrome vendors charge for compute time per page.
The point is just to stop what is effectively a DDoS because of shitty web crawlers, not to stop the crawling entirely.
I'm not sure. IP-based rate limits have a well-known issue with shared public IPs for example. Technically they are also more resource-intensive than cryptographic approaches too (but I don't think that's not a big issue in IPv4).
These are also harmful to human users, who are often behind CGNAT and may be sharing a pool of IPs with many thousands of other ISP subscribers.
Based on the comments here, it seems like many people are struggling with the concept.
Would calling Anubis a "client-side rate limiter" be accurate (enough)?
And are you going to support older browsers? I tested Anubis with https://www.browserling.com with its (I think) standard configuration at https://git.xeserv.us/xe/anubis-test/src/branch/main/README.... and apparently it doesn't work with Firefox versions before 74 and Chromium versions before 80.
I wonder if it works with something like Pale Moon.