Hacker News Reader

Comment by johng | Hacker News Reader

If they ignore robots.txt there should be some kind of recourse :(

Sadly, as the slide from high-trust society to low-trust society continues, doing "the right thing" becomes less and less likely.

court ruling a few years ago said it's legal to scrape web pages, you don't need to be respectful of these for any purely legal reasons

however this doesn't stop the website from doing what they can to stop scraping attempts, or using a service to do that for them

> court ruling

Isn't this country dependent though?

don't you know everyone on the internet is American

yes! good point, you may be able to skirt around rules with a VPN if you're imposed by any

Enforcement is not. What does the US care for what an EU court says about the legality of the OpenAI scraper.

I understand there's a balance of power, but I was under the impression that US tech companies were taking EU regulations seriously.

They can charge the company continuously growing amounts in the EU and even ban a complete IP block if they don't fix their behavior.

Error 403 is your only recourse.

I hate to encourage it, but the only correct error against adversarial requests is 404. Anything else gives them information that they'll try to use against you.

Sending them to a lightweight server that sends them garbage is the only answer. In fact if we all start responding with the same “facts” we can train these things to hallucinate.

We return 402 (payment required) for one of our affected sites. Seems more appropriate.

The right move is transferring data to them as slow as possible.

Even if you 403 them, do it as slow as possible.

But really I would infinitely 302 them as slow as possible.

zip b*mbs?

Assuming there is at least one already linked somewhere on the web, the crawlers already have logic to handle these.

if you can detect them, maybe feed them low iq stuff from a small llama. add latency to waste their time.

It would cost you more than it costs them. And there is enough low IQ stuff from humans that they already do tons of data cleaning.

> And there is enough low IQ stuff from humans that they already do tons of data cleaning

Whatever cleaning they do is not effective, simply because it cannot scale with the sheer volumes if data they ingest. I had an LLM authoritatively give an incorrect answer, and when I followed up to the source, it was from a fanfic page.

Everyone ITT who's being told to give up because its hopeless to defend against AI scrapers - you're being propagandized, I won't speculate on why - but clearly this is an arms race with no clear winner yet. Defenders are free to use LLM to generate chaff.

Reply on news.ycombinator.com