Hacker News Reader

Is it because people viewed it as Google scraping the site to make an index so that people could find the site, while the AI scraping is intended so people won’t need to visit the site at all?

View on HN

Also, google is relatively considerate when crawling.

Usually though from time to time over many many years from when G was still wet behind the ears I have had to write specific defences effectively just to deal with it.

Yeah, “Relatively” is doing a lot of heavy lifting there.

I remember people with lax security and a tags that went to deletion endpoints… we all learned a lot back in the day.

The AI apps (namely chatgpt and Claude) are evolving to display external data with widgets that will potentially drive more traffic than google has been doing for a long time. Might be worth changing focus as SEO killed google.

Kagi already behave really nicely, giving a small summary with numbered footnotes that links to where Kagi got its information from.

Which IMO shows that is already possible to do this and and it has been for a while already.

Reply on news.ycombinator.com