Hacker News Reader

Comment by sneak | Hacker News Reader

You don’t have to fend off anything, you just have to fix your server to support this modest amount of traffic.

Everyone else is visiting your site for entirely self-serving purposes, too.

I don’t understand why people are ok with Google scraping their site (when it is called indexing), fine with users scraping their site (when it is called RSS reading), but suddenly not ok with AI startups scraping their site.

If you publish data to the public, expect the public to access it. If you don’t want the public (this includes AI startups) to access it, don’t publish it.

Your website is not being misused when the data is being downloaded to train AI. That’s literally what public data is for.

View on HN

> I don’t understand why people are ok with Google scraping their site (when it is called indexing), fine with users scraping their site (when it is called RSS reading), but suddenly not ok with AI startups scraping their site.

I’m pretty open‐minded about AI, and have no visceral objection to AI scraping in theory. However… the rise of AI scrapers is the first time in twenty years of running tiny websites that my servers—all my servers, consistently—hit CPU and bandwidth usage levels far above baseline, and even brush against my hosts’ monthly limits. When I check server logs, the traffic is largely bots repeatedly trawling infinite variations of version control history pages. Googlebot never crawled these enough to become a problem; Internet Archive never crawled these enough to become a problem. But the current scrapers are, and literally everyone I talk to who hosts their own VCS webservers is having the same problem right now. Many are choosing to reduce the problem by injecting Cloudflare or Anubis, which I as a user hate, and refuse to do myself—but what alternative can I propose to them? People running these servers are just users of Gitea and CGit and whatnot, they’re not capable of rewriting the underlying tools to reduce the CPU usage of diff generation.

Seems to me like standard dumb rate limiting (max 100r/min or like 250r/5min) per IP would solve the problem easily without affecting human users at all.

Surprised this isn’t a feature in Gitea already (even though it is better done in the reverse proxy web server).

Some scummy scrapers are snowshoeing their way through scads of (residential) IPs, just for one example.

Assuming that there are simple universal solutions, eg based on IP or UA, suggests that you have been lucky enough not to deal with many varieties of this misbehaviour.

Is it because people viewed it as Google scraping the site to make an index so that people could find the site, while the AI scraping is intended so people won’t need to visit the site at all?

Also, google is relatively considerate when crawling.

Usually though from time to time over many many years from when G was still wet behind the ears I have had to write specific defences effectively just to deal with it.

Yeah, “Relatively” is doing a lot of heavy lifting there.

I remember people with lax security and a tags that went to deletion endpoints… we all learned a lot back in the day.

The AI apps (namely chatgpt and Claude) are evolving to display external data with widgets that will potentially drive more traffic than google has been doing for a long time. Might be worth changing focus as SEO killed google.

Kagi already behave really nicely, giving a small summary with numbered footnotes that links to where Kagi got its information from.

Which IMO shows that is already possible to do this and and it has been for a while already.

>I don’t understand why people are ok with Google scraping their site (when it is called indexing), fine with users scraping their site (when it is called RSS reading), but suddenly not ok with AI startups scraping their site.

Google and the publisher share the value derived from that scraping event (somewhat equitably) when a user clicks a link in Google's index. OpenAI, Anthropic, and co extract all of the value, and incur costs to the publisher, from their scraping events.

Tell me you haven’t had your site scraped by an abusive AI crawler without telling me you haven’t had your site scraped by an abusive AI crawler.

Google and traditional crawlers are fine. This new breed of AI crawlers have no scruples, no rate limits, and will just load pages and follow links as fast as they can, very likely overwhelming smaller web servers in the process.

I too have blocked many of them to avoid them filling my logs with nonexistent page accesses, for example.

I run a public Gitea at https://git.eeqj.de and while I see it getting scraped, I haven’t noticed any effects whatsoever from the additional load. No mitigations have been necessary.

The OP notes at some point that the scrapers were generating lots of source tarballs individual commits. That takes server resources, and a small server can't sustain many of those requests per second. I wonder why you (apparently) don't suffer from this — in fact one can even wonder why such tarballs are generated in the first place, it sounds very pointless.

> The OP notes at some point that the scrapers were generating lots of source tarballs individual commits.

I refuse to believe OAI has a scraper that is designed to push commits. It is obvious to me that we are conflating other forms of malicious use with the boogeyman.

They are talking about the standard “Download zip”, which can also be done for commits individually. They aren’t precomputed, which causes the load.

They don't need to be pre-computed or stored. You can directly stream stdout from git archive to the HTTP response. This is the default behavior of git archive.

And git archive uses zero cpu therefore it can handle thousands of archive requests per second. Right? Right??

Yes, your CPU should be nearly idle. Git archive is I/O bound.

The answer is attribution. also putting things public does not imply no copyright or license

Your expectations are arbitrary and you're not hosting it so they can take whatever measures they want. Just like some publish "public websites" and immediately show a "subscribe" button if you want to see more. It's their prerogative.

Reply on news.ycombinator.com