Hacker News Reader

Comment by jsnell | Hacker News Reader

Proof of work does not scale. It trades something fungible and incredibly cheap (CPU) for something incredibly expensive (user-visible latency). There is no set of parameters where the cost is going to be a meaningful deterrent to any kind of abuse (even something as low-yield as scraping) without adding crippling amounts of latency to real users.

> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.

There is no dilemma. They get a token, they maybe do some automated multi-armed bandit per-site to figure out how to maximize the extraction rate they get from a single token, and then they use an IP for that many requests / that amount of time before ditching it.

View on HN

The success of the ongoing Anubis rollout proves the opposite. People are used to slowly-loading websites - the rise of garbage SPAs has seen to that. Staring at a spinner for a second every once in a while is not an issue for genuine users.

On the other hand, the additional CPU usage rises the compute cost of scraping by several orders of magnitude. If you don't have to scrape this specific website, you'd be stupid not to move on and hit someone else.

Ideally the general cost of scraping would be high enough that it isn't affordable any more - especially anonymized - but considering the amount of money brainlessly being pumped into AI I doubt that'll happen any time soon.

You could of course also make the argument that the user's time is worth something as well and should be included as part of the cost, but that ship sailed a loooong time ago. If you care about that, you should be calling for the death of client-side Javascript and any form of advertising.

Scrapers generally aren't looking for random websites to scrape. They have a specific URL in mind. Only if the goal was DDoS would they not care which URL was accessed.

They also don't care which URL was accessed when the goal is scraping as much text as possible for AI training.

> It trades something fungible and incredibly cheap (CPU)

it could be RAM-bound, which is very much NOT cheap nowadays :)

Yes, but the people with the RAM nowadays are the data centers, not the end users.

Sure, but end users as a group still have a significant amount of RAM.

Even on a low-specced machine you can afford to have the currently-active website tab consume a few hundreds of megabytes of RAM. It was mostly sitting idle anyways, so most people aren't even going to notice. Multiply that by a few thousand concurrent visitors and you're burning hundreds of gigabytes without anyone caring.

Some scraper being forced to install hundreds of gigabytes of extra RAM in their crawling node? They will notice having to go from a $0.50 / h instance to a $15.00 / h one, solely for the extra RAM requirement.

It doesn't make the economics any different. In a browser environment, you're maybe looking at the acceptable lease being 100MB for 1 second. Much more than that, and you start hitting limits of what browsers will let you do on low-end phones. Longer than that, and we're back to the user-observable latencies being too long.

100MB for 1 second just is not much of a deterrent.

Reply on news.ycombinator.com