Hacker News Reader

Comment by blakesterz | Hacker News Reader

If normal crawlers are a light rain, AI crawlers are a hurricane. Most sites can handle some rain, but they are not built to handle hurricanes. AI crawlers can look like DDOS attacks. The worst offenders will just crawl a site as fast as possible until it goes offline.

View on HN

Yes, I understand. And sorry to hear that. But I'm trying to understand how it is related to AI. How come this is happening with AI crawlers but not with traditional web index crawlers. If the pattern is so common (which is confirmed by multiple credible sources) there must be some interesting and potentially useful explanation.

https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-th...

This is interesting on the subject. Maybe gives you some understanding. Posted on HN a few minutes ago.

My understanding is that everyone wants to be first in the AI race, so they throw all the rules everyone else agreed on overboard.

Search engines link to websites. They want the websites up, so its worth a little extra work to avoid harming them. LLMs seek to replace the websites.

Search engine crawlers are more mature and better written.

I suspect a lot of LLM crawling an development is done under time pressure to get things done while the investors money is still coming in to fund it. DO stuff in a hurry, and it will be less competently done.

Ordinary search indices don't contain the entire target site, while LLM-style so called AI does consume it all. I would guess some of these crawlers are subcontractors rather than "AI" companies, i.e. they compete on having the most complete and fresh dataset you could rent for "training".

Whenever the market decides the Internet is too full of slop to be usable for "training" the one that has the most copies of the pre-"AI" Internet wins. Some of the traffic is likely "AI" "tool use", i.e. bot scraping as part of running some LLM, i.e. "AI" "research".

The big scraping bots have gone from stupid to ruthless. Previously it was irritating that some of them got stuck traversing cyclical link paths on your site or on-the-fly generated pages, now it's like your silly family blog suddenly got very popular for no good reason and it puts a lot of load on the tiny amount of hardware it's served from.

Reply on news.ycombinator.com