Hacker News Reader

Comment by ranger_danger | Hacker News Reader

> In retrospect implementing the proof of work challenge may have been a mistake

Why?

What would the alternative have been?

It does two things: Force everyone (including scrapers) to run a real JS engine, and force everyone to solve the challenge.

The first effect is great, because it's a lot more annoying to bring up a full browser environment in your scraper than just run a curl command.

But the actual proof of work only takes about 10ms on a server in native code, while it can take multiple seconds on a low-end phone. Given the companies in questions are building entire data centers to house all their GPUs, an extra 10ms per web-page is not a problem for them. They're going to spend orders of magnitude more compute actually training on the content they scraped, than solving the challenge.

It's mostly the inconvenience of adapting to Anubis's JS requirements that held them back for a while, but the PoW difficulty mostly slowed down real users.

I'd have thought that anyone doing serious scraping has been driving a browser for years at this point.

you can even get a curl fork that drives a browser under the hood

It does 2.5 things actually - it counters the aggressive use of proxies by scrapers.

Without getting into the alternatives: scraper defense isn't a viable proof of work setting, because there's no asymmetry to exploit. You're imposing exactly the same cost on legit users as you are on scrapers. Economies of scale mean that the marginal cost for your adversary is actually significantly lower than for your real users.

What the Anubis POW system is doing right now is exploiting the fact that there's been no need for crawlers to be anything but naive. But the cost to make them sophisticated enough to defeat the POW system is quite low, and when that happens, the POW will just be annoying legit users for no benefit.

I don't know if "mistake" is the word I'd use for it. It's not a whole lot of code! It's a reasonable first step to force crawlers to emulate a tiny fraction of a real browser. But as it evolves, it should evolve away from burning compute, because that's playing to lose.

Wait, but there is an asymmetry. Legitimate user spends at least a dozen seconds on a page, they don't care about 10ms overhead. For a scraper, however, 10ms overhead can easily be 10x the time it spends on a page overall - the scraper is now ten times slower.

However the exact PoW implementation (hash) chosen by Anubis might significantly reduce this asymmetry, because the calculation speed is highly dependent on hardware.

No, I don't think this is accurate. You have to look at both the cost and the benefit. If you're an AI scraper, it's literally just "what does the marginal next token of training data cost me" --- the answer is: the same as the marginal next token of content costs a reader.

Tavis Ormandy went into more detail on the math here, but it's not great!

I don’t understand what you mean. Training an LLM requires orders of magnitude more tokens than any one human will ever read. Perhaps an AI company can amortize across all their users, but it would still represent a substantial cost. And I’m pretty sure the big AI companies don’t rely on abusive scraping (i.e. ignoring robots.txt), so the companies doing the scraping may not have a lot of users anyway.

Tavis Ormandy's post goes into more detail about why this isn't a substantial cost for AI vendors. For my part: we've seen POWs deployed successfully in cases where:

(1) there's a sharp asymmetry between adversaries and legitimate users (as with password hashes and KDFs, or antiabuse systems where the marginal adversarial request has value ~reciprocal to what a legit users gets, as with brute-forcing IDs)

(2) the POW serves as a kind of synchronization clock in a distributed system (as with blockchains)

What's case (3) here?

The next word is worth less to AI scrapers than to human readers - AIs need to read thousands of articles to get as much value as a human gets from one good article. If you make it cost, say, 5c-equivalent to read an article (but without the overhead of micropayments and authorisations), human readers will happily pay that whereas AI scrapers can't afford even 1c-equivalent.

They care about whether the rewards exceed the costs; they don't give a shit what the actual cost is.

If it costs them $1000 to grab a web page but they earn $1001 then they will do that again and again to earn that buck.

I don't know of any network latency <=1ms over the public internet, so 10ms overhead might be 2x at best.

> Legitimate user spends at least a dozen seconds on a page, they don't care about 10ms overhead.

Unfortunately for the user on a low-end phone, the overhead can be several seconds. For the scraper it's only ever 10ms because that's running on a (relatively) powerful server CPU.

The scraper unlike a legitimate human can load and analyse parallel websites simultaneously, so really the difference makes no difference to a scraper.

Say a user browses 10 sites, all restricted by Anubis that add 5 second to the load time, that's 50 additional seconds the user is spent waiting. A scraper with enterprise grade server hardware? that's 5 seconds for all 10 sites.

An unavoidable aspect of abuse problems is that there is no perfect solution. As the defender, you’re always making a precision vs. recall tradeoff. After you’ve picked off the low hanging fruit, most of the time the only way to increase recall (i.e. catch more abuse) is by reducing the precision (i.e. having more false positives, where a good user is falsely considered an abuser).

In an adversarial engineering domain neither the problems or solutions are static. If by some miracle you have a perfect solution at one point in time, the adversaries will quickly adapt, and your solution stops being perfect.

So you’ll mostly be playing the game in this shifting gray area of maybe legit, maybe abusive cases. Since you can’t perfectly classify them (if you could, they wouldn’t be in the gray area), the options are basically to either block all of them, allow all of them, or issue them a challenge that the user must pass to be allowed. The first two options tend to be unacceptable in the gray area, so issuing a challenge that the client must pass is usually the preferred option.

A good counter-abuse challenge is something that has at least one of the following properties:

1. It costs more to pass than the economic value that the adversary can extract from the service, but not so much that the legitimate users won’t be willing to pay it.

2. It proves control of a scarce resource without necessarily having to spend that resource, but at least in such a way that the same scarce resource can’t be used to pass unlimited challenges.

3. It produces additional signals that can be used to meaningfully improve the precision/recall tradeoff.

And proof of work does none of those. The last two by construction, since compute is about the most fungible resource in the world. The last doesn't work since it's impossible to balance the difficulty factor such that it imposes a cost the attacker would notice but would be acceptable to the defender.

If you add 10s to the latency for your worst-case real users (already too long), it'll cost about $0.01/1k solves. That's not a deterrent to any kind of abuse.

So proof of work just is a really bad fit for this specific use case. The only advantage is that it is easy to implement, but that's a very short term benefit.

In practice, any automated work that a real user is willing to wait through will be trivial to accomplish for an organization which scrapes the entire Internet. The real weight behind Anubis is the Javascript gate, not the PoW. It might as well just fetch() into browser.cookies.set().

They also suggest maybe “proof of React” would be better with a link to this rough proof of concept:

https://github.com/TecharoHQ/anubis/pull/1038

Could someone explain how this would help stop scrapers? If you’re just running the page JS wouldn’t this run too and let you through?

Low-effort scrapers don't run JS, they just fetch static content.

But then they couldn’t get past the current Anubis. Sonia the idea it would just be cheaper for clients?

That's the idea. Impose software requirements on the client instead of computational requirements.

They admitted that this was a 'shitpost'.

> how this would help stop scrapers

I think anubis bases its purpose on some flawed assumptions:

- that most scrapers aren't headless browsers

- that they don't have access to millions of different IPs across the world from big/shady proxy companies

- that this can help with a real network-level DDoS

- that scrapers will give up if the requests become 'too expensive'

- that they aren't contributing to warming the planet

I'm sure there does exist some older bots that are not smart and don't use headless browsers, but especially with newer tech/AI crawlers/etc., I don't think this is a realistic majority assumption anymore.

In part because this particular proof of work is absolutely trivial at scale, with commercial hardware able to do 390TH/s, while your typical phone would only be able to do a million and still have acceptable latency.

Reply on news.ycombinator.com