Hacker News Reader

Comment by ammario | Hacker News Reader

There are PoW approaches that even the playing field between data centers and desktops. RandomX is my favorite.

For what it's worth I'm working on hashx support. It's just going to take a bit to ship while I do browser testing with broken browser configs.

Interesting. How do they tell the difference between legitimate and forged ip owner records?

It's not about traffic identification at all, but rather a hashing algorithm that is deliberately resistant to parallelization and GPU/ASIC acceleration, which shrinks the gap in solving speed between the fastest systems (i.e. datacenter-class compute resources) and typical systems (e.g. the CPU in your smartphone or laptop).

Uh, is it resistant to parallelization across multiple sites? Because that's the situation for the scrapers. They're not trying to solve a single PoW challenge across many cores.

Typically, the machine doing the content processing, including solving PoW, is the centralized "control" node described in the article, not the machines who's IP addresses are being used. In typical residential proxy networks, the residential proxies are exposed to the customer (the person paying for and using the proxies) as just SOCKS5 addresses, and no computational power from those compromised devices is made available for the scraper besides that used to power the SOCKS5 server itself, the customer is just paying for the transport and address (and indeed, is often billed on either a per-GB or per-IP basis).

In effect, if the customer (the entity paying for and using the proxies) wants to solve PoW challenges through those connections, it is indeed the customer who must pay that compute cost, not the compromised devices.

Note that this is the case for a majority of, but not all, residential proxy networks, which often are built through quasi-voluntary distribution channels, including SDKs included in otherwise legitimate mobile applications distributed through Apple's App Store and Google Play.

These distribution channels tend to be categorically unavailable (or at least unreliable) for true RAT-style malware that enables remote operators to dynamically assign arbitrary computational workloads to client devices.

This isn't to say that true botnets built with actual malware delivered through either software exploits, phishing attacks, or watering hole attacks don't also perform as residential proxy networks, but such categories are a relatively small subset of all residential proxy networks, and there are much higher ROI malicious activities to be performed on these devices rather than serving as relatively mundane traffic networks for scraping.

That's a completely different question, your claim was about parallelism.

Ah, I see what you're getting at. Yes. You can think of any given computer aa having a fixed amount of compute budget for these types of acceleration-resistant hashing algorithms. Let's say the scraper can perform 10,000 hashing operations per second total on their machine, and needs an average of 1,000 hashing operations to solve the PoW. It's a minor detail, but note that these PoW challenges non-deterministically vary in the number of hashing operations needed to produce a valid hash, not dissimilar to bitcoin mining, where a hash with a certain number of 0s prefixed is sought, and the scraper essentially has to brute force through all possible inputs until an input that produces a valid hash is found.

In a well-designed PoW systems, there is a per-site prefix or suffix that is required to be prepended or appended to these random inputs, and it may change not only between websites, but even between PoW sessions on the same website, and should not be predictable - only being disclosed to the client at the time the PoW challenge is issued. In such a case, the scraper cannot simply precompute a bunch of valid hashes that work across multiple sites, nor a bunch of valid hashes that will always be good for even one site, the scraper operator will need to compute these hashes (with a limited budget to do so) upon initiating each PoW session.

So, in your example, each request needs 0.1 core-seconds of PoW. Is that right? Or, since you're saying 10000 hashes across the entire machine and 1000 needed, then on a typical 96-core server you need 9.6 core-seconds of PoW? The former means you pay about $0.0000005 per request at standard cloud rates; the latter means you pay about $0.00005 per request _and_ your site is totally unusable by legitimate clients. Both are _easily_ worth it for someone backed by VC billions and hungry for data. The network and storage fees are likely to be more significant than that already, not to mention actually training a model; they don't balk at downloading terabytes of crap already.

Note that none of this assumes any sort of acceleration from parallelization (be it through GPUs or reusing work across servers) or precomputation relative to what a normal client does. Compute is just really cheap in dollars compared to the cost of having a user wait, and these companies _also_ have a lot of appetite for spending dollars compared to that of a normal user. As others have pointed out, the only reason why Anubis works (sort-of; not for everyone) right now is that it is uncommon enough, essentially “proof that you bothered to have your crawler run JavaScript at all”. It's a confusion measure.

Proof of work does not work.

You raise some good points here, but PoW is meant to be one tool in the toolbox, not the only line of defense. You can still maintain blocklists of known scrapers (or better yet, have your PoW system be aware of them and silently adjust the difficulty to an impossible level, such that the scraper gets stuck trying to solve your PoW challenge until it hits a timeout configured by the scraper, if they were wise enough to configure one). It's also courteous to not only build and maintain your own blocklists, but to share them with e.g. vtotal and spamhaus, to help protect others.

Similarly, you have tarpits, which generate infinite mazes of garbage data, or even deliberately poisoning training data (should the scrapers be training LLMs) though this don't entirely eliminate the deleterious effects of scraping on the host's web server (more info: https://arstechnica.com/tech-policy/2025/01/ai-haters-build-...).

If the premise was evaluating whether or not PoW would be a magic silver bullet that stops scrapers all by itself, then you are correct, it does not stop all scrapers. Scraping and anti-scraping is fundamentally a constantly evolving cat and mouse game that demands adaptability and punishes complacency from all participants trying not to lose.

Reply on news.ycombinator.com