Ask HN: How do I save my content from AI crawlers? | Hacker News Reader

Ask HN: How do I save my content from AI crawlers?

8 pointssrameshc24 comments

I am building something where we (a very small team) are creating content the old fashioned way and for each content it takes us a lot of effort. If I wanted to be publicly viewable but I am equally worried about AI crawlers now. Is putting the content behind auth the only way or are there other means ?

View on HN

The only way to prevent crawlers from scraping your content is to make it unavailable to them. I am not suggesting any clever application or code trick. When I say unavailable I mean a separate network. Network isolation is ideal but there is also VPN intranet.

You cannot have both public access content and yet limit it to limited parties at your discretion. The content is either public or it’s not and obfuscation, even via authentication, is not a security solution. The role of authentication is not to limit access but to segment access such that account specific content is not intermingled between accounts.

> Is putting the content behind auth the only way

No because I can still copy and redistribute your auth-locked content. By writing something and distributing that writing you are making it possible to train on the contents. That's the way things are now.

Add a reasonable fee in ToS. Something like $100 per article, and $5 per article per year for continued use in a model.

It may discourage some companies from scrapping your content.

And if they ignore the license, you may have grounds to collect the fee if it becomes obvious that they are using your content in the AI models. (Unlikely to collect though)

This raises the question: what would be a fairly affordable way to detect and prove scraping has occurred?

There is some ongoing research https://github.com/computationalprivacy/copyright-traps

Considering big companies like Cloudflare and Google have teams doing this 24/7, I’d say it’s impossible to track a stealth individual behind proxies.

Check logs for the AI user agent?

Disallow them in robots.txt. Visit https://[originating IP] and opt out. Email abuse@[owner of their IP]. It would be fun to distinguish an AI crawler from a regular crawler, btw.

AI companies have chosen to either ignore robots.txt entirely or they just rename their crawler so it technically doesn't apply to them.

[0] https://www.businessinsider.com/openai-anthropic-ai-ignore-r...

[1] https://www.404media.co/email/87e0e07a-7d24-4788-b417-1821b4...

I wanted to serve content to a regional area but avoid discovery by spammers and wasting bandwidth on crawlers. I didn’t need SEO as I had a separate channel to notify users my site existed. At first I thought I’d blacklist the bots: no Google, no bing, no OpenAI. But I realised that a whitelist would be much more effective.

The process I used was to gather the ASNs of every local ISP where my users are. Thanks to BGP looking glass services, you can convert this to a list of every advertised prefix. You can coalesce this into an optimised data structure using “ipset” to build a firewall rule that affects only addresses you whitelist.

I also found a data center offering colocation which is directly peered with every major ISP in my area and a few of the minor ones.

Any visitor that doesn’t match my whitelist sees a decoy page. For the protected content, I set IP TTL just high enough to deliver only within the service area.

One drawback of this approach is it’s difficult to include local mobile users (without including their entire parent network) because they will need higher TTLs and come from a network with many prefixes. I’d be interested in an iptables rule that can drop connections after a few packets based on an RTT heuristic.

This approach is the opposite of “zero-trust” and it fails as soon as your threats start moving into into your user’s network.

perhaps adding a challenge (captcha, other) if the pattern of the client is being considered malicious (requests per min, avg time on page, other stats)? This might not entirely block the crawler, but will still provide data that will allow blocking entirely the source if it's a cloud IP range, degrading the velocity of crawling if measures taken. The other way around would be displaying publicly a degraded in any way copy of the content and delivering a full (high quality) one after signing in.

Overall - simply making life a bit harder for the crawler (AI or plain old hardcoded one).

https://darkvisitors.com/agents

I saw a saas recently that has something targetting this. They will automatically update your robot.txt to include different crawlers they find.

This will only stop crawlers that obey robots.txt though.

Wouldn't it be possible to block IP ranges that belong to known cloud providers? Normal people using the browser don't have such IP address assigned. You would be blocking as well other kind of visitors (scrappers and the like), but I guess that's a fair price to pay.

A lot of shady scrapers run on residential IP addresses.

Let's forget about the small shady scrapers. What about OpenAI? My bet is that they run their scrapers in Azure/AWS/GCP. Let's ban the big cloud providers.

I just thought about an architecture that would be able to prevent access from AI crawlers while still public for anyone to access.

What would be the demand for something like this as a SaaS? Wondering if it would be worth developing it.

No surefire way, same as blocking any other poorly-behaved crawlers. You can use robots.txt, Cloudflare, CAPTCHAs, IP range blocks, etc., but at the end of the day none of those are foolproof. If someone really wants to scrape you, they will.

> If someone really wants to scrape you, they will.

I don't think that's a problem. What you said is right, but no giant (like OpenAI or Google) will employ resources to scrape YOUR website. They do everything at scale, so it should be enough to block the big players (until they find a way to bypass the block, which it may take them time and money)

I really don't think there is any solution to this right now. This problem makes the web unviable for me as a publication/distribution platform.

i often browse the web for fun (but in a careful way), I can totally see how changing up the CSS class names in the site’s HTML regularly would mess up a bunch of the XPATH/CSS selectors in the crawler. It’d seriously be a nightmare for me if the site owners could just flip a 'switch' and change the class names easily

If you’re scraping for text, you can always render the page as an image and do OCR on it, worst case.

Hope companies obey robots.txt.

They don't. Also, they use user agent strings used by other bots (e.g. Googlebot).

I hate this answer but have to agree with it. That we are in a position where small content creators are at the mercy of giants who regularly ignore the rules tells me the Internet is a lost cause. I am not even sure how we start to take back whatever power we can and think about this every time I publish an article to my tiny tech blog, which is super discouraging.

Reply on news.ycombinator.com