Hacker News Reader

Comment by mootothemax | Hacker News Reader

> The anubis author has stated they recognize it's an arms race, but PoW scales.

The scraper wars are largely between script kiddies and people with both deep intimate networking and DOM knowledge. Yes greyhairs, I’m looking at you.

The problem is, you can’t PoW every page load and resource request because the user experience will suck and people will run away. And that window - the gap between what people will tolerate vs draconian enforcement - is exactly what the scrapers exploit.

And looking at the PoW options out there - I’ve seen at least one PoW WAF (honestly can’t remember if azure or amazon) have their PoW boil down to repeated trigonometric functions, ie very optimisable.

It’s a neat concept, but the answer and future to my eyes look bleak.

View on HN

Oh, but you can PoW every page.

Your typical end user doesn't switch IPs that often, so it's fine to Anubis them again when they do. A scraper, on the other hand, has a tradeoff to make between rotating ips often (requiring a challenge on every request) or keeping only a few IPs (making cross-request identification much more valuable and reliable).

> Oh, but you can PoW every page.

They meant you can’t PoW every page transition.

If clicking every link on your website throws you back to another Anubis page for 2-3 seconds, users will bounce.

That’s why Anubis does an up front challenge and then you’re good for a while. It’s a really low cost for the scrapers.

> That’s why Anubis does an up front challenge and then you’re good for a while. It’s a really low cost for the scrapers.

Except that doing hundreds of requests from the same IP makes it pretty trivial to detect scraping, opening you up to being banned, or fun stuff like a slowloris or being fed poisoned data.

There are major websites that take more than 2-3 seconds to load every new page. Google, for example. Reddit. Facebook. Instagram. Doesn't seem to have hurt them.

Anubis's default 1-week token lifetime may not be nearly enough to dissuade enough scraper networks to make a difference, particularly with the default weight->difficulty level hierarchy, but that's for individual site admins to determine.

We can all argue based on how we envision "ideal" scraper networks being run and whether the web-PoW concept would stand up to that. However, what matters at present is that anubis helps many sites cope with misbehaving bot scrapers written by the script kiddies you mention, who don't care if the internet burns as long as they finish their scrape 1 hour faster. If anubis motivates them to devote a few brain cells to make their scrapers smarter, they may also fix the scrapers to not take down the sites they're scraping.

One of the ideas behind Anubis was to incentivize a scraper to stop hiding, because every change of identity brings another challenge page.

Reply on news.ycombinator.com