Hacker News Reader

Comment by tedunangst | Hacker News Reader

> It remains unclear why these companies don't adopt more collaborative approaches and, at a minimum, rate-limit their data harvesting runs so they don't overwhelm source websites.

If the target goes down after you scrape it, that's a feature.

View on HN

This has me wondering what it would take to do a bcrypt style slow hashing requirement to retrieve data from a site. Something fast enough that a single mobile client for a user wouldn't really feel the difference. But an automated scraper would get bogged down in the calculations.

Data is presented to the user with multiple layers of encryption that they use their personal key to decrypt. This might add an extra 200ms to decrypt. Degrades the user experience slightly but creates a bottleneck for large-scale bots.

Check out Anubis - it's not quite what you're suggesting but similar in concept: https://anubis.techaro.lol/

How does it work? I don’t have time to read the code, and the website/docs seem to be under construction.

Does it have the client do a bunch of SHA-256 hashes?

Yeah it's like hashcash. You have to try random numbers until you roll a hash with enough leading zeroes. Then you get a cookie (JWT I think) that's valid for a week.

SHA2 can run on ASICs and isn't memory-hard, so I'm hoping someone will add something tougher

Interesting! Not how I'd approach it but certainly thinking along the same lines.

I just hit a site with this --- and hit the back button immediately.

Better than a 500 error

Also better than a third party service IMO because of the privacy implications. You could even potentially give users the choice (complete overkill but technically you could do it).

Maybe once you can use something more professional as an interstitial page

From the announcement page: https://xeiaso.net/blog/2025/anubis/

> RPM packages and unbranded (or customly branded) versions are available if you contact me and purchase commercial support. Otherwise your users have to see a happy anime girl every time they solve a challenge. This is a feature.

I'm going to be making distro packages and binaries public. I will have to figure out white label monetization I guess.

It's open source with MIT license, so you should be able to remove those images yourself if you want.

If we are able to detect AI scrapers then I would welcome a more strategic solution: feed them garbage data instead of the real content. If enough sites did that then the inference quality would take a hit and eventually the perpetrators, too.

But of course this is the more expensive option that can't really be asked of sites that already provide public services (even if those are paid for by ads).

I really don’t think we have such a lack of misinformation that we need to invest in creating more of it, no matter the motive.

The problem is that the server also has to do the work. Fine for an infrequent auth challenge. Not so fine for every single data request.

Tons of problems are easier to verify than to solve.

do any of them involve mining bitcoin?

sure

Maybe there is a way for the server to ask the client to do the work?

Something similar to proof-of-work but on a much smaller scale than Bitcoin.

just add some delay to your response, we don't have to waste any more energy on meaningless calculation.

Adding delay means you have to keep more connections open at a single time. Parallelism doesn't favor a server if your problem is already a small server getting hit by a big scraper

How expensive is it to just keep a connection open?

About 20 kilobytes of socket + TLS state, if you've really optimised it down to the minimum. Most server software isn't that lean, of course, so pick a framework designed for running a million or so concurrent connections on a single server (i.e. something like Nginx)

Right it would need an algorithm with widely different encryption speeds vs decryption speeds. Lattice-based cryptography maybe?

Hash functions are all you need.

Yeah, searching for hashes with some prefix is easy to set up.

Companies running those bots have more than enough resources

Nobody has unlimited resources. Everything is a cost-benefit analysis.

For highly valuable information, they might throw the GDP of a small country at scraping your site. But most information isn't worth that.

And there are a lot of bad actors who don't have the resources you're thinking of that are trying to compete with the big guys on a budget. This would cut them out of the equation.

Make all websites intentionally waste energy as a strategy to defeat unscrupulous operators has negative costs and marginal benefits.

Resources applied to prevent bad actors from degrading or destroying the commons has always been the cost of civilization.

Then use it to mine monero or similar.

The idea that you should pay for content shouldn't be an insane pipedream. It should be the default on the internet.

Maybe then we wouldn't be in the situation where getting new users is an existential threat to the majority of websites.

I'll add a link on my site recommending that visitors petition their elected officials for a pollution tax

Why? What is the goal of a scraper, and how does disabling the source of the data benefit them?

> Why? What is the goal of a scraper, and how does disabling the source of the data benefit them?

The next scraper doesn’t get the data. People don’t realize we’re not compute limited for ai, we’re data limited. What we’re watching is the “data war”.

at this point we're _good data_ limited, which has little to do with scraping.

Why kind of data that isn’t public would be so valuable for AI training?

Seems like there’s a fuck ton. All of Wikipedia, GitHub for code, etc.

I can understand targeting certain sites like Reddit, etc. but not random websites

It's to rip off copyrighted content and profit from it instead of the original authors. It's like every other low rent and highly automated scam that finds it's way onto the internet.

If you look closely even Google does this. This is probably why many popular sites started getting down ranked in the last 2 years. Now they're below the fold and Google can present their content as their own through the AI box.

Please remember that Google only needs to be marginally better than the competition. And, of course, their primary biz is ads, not serving great results; that is a distant second priority.

Their biz is ads, but since search is winner takes all they need only be marginally better than the competition... twenty years ago.

> Their biz is ads,

Yea, but, the FTC doesn't want it to be.

Discord I guess would be quite valuable, even the de facto public servers.

Honestly it's hard to tell how much more value the LLM people are going to get out of another copy of the internet.

It feels a lot like they're stuck for improvements but management doesn't want to hear it.

It's a bit strange to talk about stuck when the most recent breakthrough is less than a year old.

I’m not sure what you mean by breakthrough, but if you’re talking about Deepseek, it’s more of an incremental improvement than a breakthrough.

Scraping social media is good data, even without ML. The fact that something is "happening" to people in a social space inherently has importance to people. The specter of law is more threatening to whether companies can get their hands on good data.

Now the only way to obtain that information is through them

I guess one could make a point that competition will no longer have the access to the scraped data.

Reply on news.ycombinator.com