Hacker News Reader

Comment by emseetech | Hacker News Reader

This has me wondering what it would take to do a bcrypt style slow hashing requirement to retrieve data from a site. Something fast enough that a single mobile client for a user wouldn't really feel the difference. But an automated scraper would get bogged down in the calculations.

Data is presented to the user with multiple layers of encryption that they use their personal key to decrypt. This might add an extra 200ms to decrypt. Degrades the user experience slightly but creates a bottleneck for large-scale bots.

View on HN

Check out Anubis - it's not quite what you're suggesting but similar in concept: https://anubis.techaro.lol/

Interesting! Not how I'd approach it but certainly thinking along the same lines.

I just hit a site with this --- and hit the back button immediately.

Better than a 500 error

Also better than a third party service IMO because of the privacy implications. You could even potentially give users the choice (complete overkill but technically you could do it).

Maybe once you can use something more professional as an interstitial page

From the announcement page: https://xeiaso.net/blog/2025/anubis/

> RPM packages and unbranded (or customly branded) versions are available if you contact me and purchase commercial support. Otherwise your users have to see a happy anime girl every time they solve a challenge. This is a feature.

I'm going to be making distro packages and binaries public. I will have to figure out white label monetization I guess.

It's open source with MIT license, so you should be able to remove those images yourself if you want.

How does it work? I don’t have time to read the code, and the website/docs seem to be under construction.

Does it have the client do a bunch of SHA-256 hashes?

Yeah it's like hashcash. You have to try random numbers until you roll a hash with enough leading zeroes. Then you get a cookie (JWT I think) that's valid for a week.

SHA2 can run on ASICs and isn't memory-hard, so I'm hoping someone will add something tougher

If we are able to detect AI scrapers then I would welcome a more strategic solution: feed them garbage data instead of the real content. If enough sites did that then the inference quality would take a hit and eventually the perpetrators, too.

But of course this is the more expensive option that can't really be asked of sites that already provide public services (even if those are paid for by ads).

I really don’t think we have such a lack of misinformation that we need to invest in creating more of it, no matter the motive.

The problem is that the server also has to do the work. Fine for an infrequent auth challenge. Not so fine for every single data request.

Maybe there is a way for the server to ask the client to do the work?

Something similar to proof-of-work but on a much smaller scale than Bitcoin.

just add some delay to your response, we don't have to waste any more energy on meaningless calculation.

Adding delay means you have to keep more connections open at a single time. Parallelism doesn't favor a server if your problem is already a small server getting hit by a big scraper

How expensive is it to just keep a connection open?

About 20 kilobytes of socket + TLS state, if you've really optimised it down to the minimum. Most server software isn't that lean, of course, so pick a framework designed for running a million or so concurrent connections on a single server (i.e. something like Nginx)

Right it would need an algorithm with widely different encryption speeds vs decryption speeds. Lattice-based cryptography maybe?

Hash functions are all you need.

Yeah, searching for hashes with some prefix is easy to set up.

Tons of problems are easier to verify than to solve.

do any of them involve mining bitcoin?

sure

Companies running those bots have more than enough resources

Nobody has unlimited resources. Everything is a cost-benefit analysis.

For highly valuable information, they might throw the GDP of a small country at scraping your site. But most information isn't worth that.

And there are a lot of bad actors who don't have the resources you're thinking of that are trying to compete with the big guys on a budget. This would cut them out of the equation.

Make all websites intentionally waste energy as a strategy to defeat unscrupulous operators has negative costs and marginal benefits.

Resources applied to prevent bad actors from degrading or destroying the commons has always been the cost of civilization.

Then use it to mine monero or similar.

The idea that you should pay for content shouldn't be an insane pipedream. It should be the default on the internet.

Maybe then we wouldn't be in the situation where getting new users is an existential threat to the majority of websites.

I'll add a link on my site recommending that visitors petition their elected officials for a pollution tax

Reply on news.ycombinator.com