Hacker News Reader

Comment by jdlshore | Hacker News Reader

Hmm, I don’t think I agree. The author is claiming that Anubis is meant to stop individuals using LLMs, but I don’t think that’s its purpose. I believe its purpose is to reduce mass scraping of data that puts excessive load on systems. It does that by increasing the cost of scraping, not by preventing it entirely.

Specifically, bad actors were ignoring robots.txt and rotating IPs to make blocking difficult. Anubis serves to make new connections more expensive, so that people will reuse a connection/cookie, and then falls back to normal means of preventing bad actors.

(That mass scraping is the result of AI training companies, yes, but it’s the mass scraping that’s the problem, not the LLMs.)

View on HN

There's plenty of arguments that mass scrapers have compute to spare but it seems to me that if Anubis makes it 100x more expensive to scrape then, for any given scraping budget, that means you get scraped 100x less. Which is the difference between your server buckling under the load or continuing to serve reliably.

If compute isn’t the bottleneck then it wouldn’t make it 100x more expensive to scrape. Of course if it does, then it’ll be effective.

In my experience, not really. Each asset is made by a new proxy, eg some TV somewhere, they send every new request via a new IP address, which is a new machine, and it doesn’t matter how long the response takes, they have already rotated to the next ip immediately and sent another request. Most of these providers have millions of IPs, and it generally doesn’t take millions of requests to scrape a website (unless it’s really big!)

And each new IP triggers a new Anubis challenge. Isn't it great?

So, once the challenges are solved by the residential proxies (with optimized native implementations [1]). Some unsuspecting TV owner is paying with their electricity bill (not the companies doing the scraping) and we all suffer through challenges.

Yes, great outcome! </s>

[1] https://lock.cmpxchg8b.com/anubis.html

Proxy networks just proxy. They don't let their client run compute on the proxy. That isn't even a possibility in socks5 protocol.

It does it by blocking the scrapers because they are really stupid scrapers. go-away blocks them too, by seeing if they load images, a much quicker test.

Except it does not actually increase the cost of scraping meaningfully. Compute is really cheap. The compute for minting an Anubis cookie will cost less than a thousandth of a cent even assuming the attacker uses the same JS implementation of proof of work rather than an optimized native implementation. That cookie can then be used for hundreds of requests.

How big of a deterrent is a millionth of a cent per page going to be? A non-existent one. Just the bandwidth from a residential proxy will cost the scraper orders of magnitude more.

Proof of work just isn't a viable counter-abuse challenge, even for something as low-yield as scraping. (It might be economically viable as a counter against some types of DDoS attacks, since they're even lower yield. But in practice it just moves the attack surface to the proof of work validation service.)

> How big of a deterrent is a millionth of a cent per page going to be? A non-existent one.

In practice, this would seem to be false, according to the admins of said sites.

Your theoretical counterargument falls apart by the reality of just putting up anubis and comparing the before and after. I don't understand why this argument shows up in every thread about anubis. There are plenty of people and orgs who have empirical before and after results. We don't need theoretical arguments when there exists actual data.

Captchas worked for a few years too. Then computers got better at solving them than humans, and we're still dealing with the fucking useless things decades later.

It's not because of the PoW though, it's just because the scraper doesn't run JavaScript. The alternative package called go-away does these tests without the PoW.

The exact claim I was replying to "It does that by increasing the cost of scraping, not by preventing it entirely." In that context, quantifying the cost isn't theoretical nitpicking!

Anubis works to the extent it does because in counter-abuse security by obscurity tends to be the best security of them all. Right now Anubis isn't popular enough for many scrapers to try working around, so they don't. But they will pour resources into working around Turnstile and Recapcha. In this regime, the exact challenge is irrelevant. Anubis would be just as effective if the challenge was running a javascript function to add two numbers instead of a proof of work.

Proof of work is just a uniquely dismal basis for a counter-abuse challenge.

Proof of work is fine if you can make it demonstrably take a few seconds. If 100 IPs are scraping you as fast as they can, making every 100ms request take 2.1 seconds means you're getting hit with 20x less traffic per unit time.

falls apart by the reality of just putting up anubis and comparing the before and after. [...] We don't need theoretical arguments when there exists actual data.

And the data says it stopped working for some sites that actually have a lot of data.

We apologize for a period of extreme slowness today. The army of AI crawlers just leveled up and hit us very badly. [...] It seems like the AI crawlers learned how to solve the Anubis challenges. [...] However, we can confirm that at least Huawei networks now send the challenge responses and they actually do seem to take a few seconds to actually compute the answers. It looks plausible, so we assume that AI crawlers leveled up their computing power to emulate more of real browser behaviour to bypass the diversity of challenges that platform enabled to avoid the bot army.

https://social.anoxinon.de/@Codeberg/115033790447125787

Excellent! When I put Anubis in front of my home Forgejo server, it blocked about 600,000 IO-expensive queries a day. I’d love to think it’s costing some moron $2000 to scrape my site the most idiotic way possible instead of just running git clone and analyzing it to their heart’s content.

There is no distinction between the two. If one can do it the other can just as easily. I know multiple people right now that can scrape any a site very easily.

I do this at scale and it took me very little time to set up and almost no resistance. So anyone who thinks that this is difficult or you're preventing people from doing this at scale, you're wrong.

Reply on news.ycombinator.com