Hacker News Reader

Comment by altdataseller | Hacker News Reader

If they follow robots.txt, OpenAI also has a bot blocking + data gathering problem too: https://x.com/AznWeng/status/1777688628308681000

11% of the top 100K websites already block their crawler, more than all their competitors (Google, FB, Anthropic, Perplexity) combined

View on HN

It's not just a problem for training, but the end user, too. There are so many times that I've tried to ask a question or request a summary for a long article only to be told it can't read it itself, so you have to copy-paste the text into the chat. Given the non-binding nature of robots.txt and the way they seem comfortable with vacuuming up public data in other contexts, I'm surprised they allow it to be such an obstacle for the user experience.

That’s the whole point. The site owner doesn’t want their information included in ChatGPT—they want you going to their website to view it instead.

It’s functioning exactly as designed.

It's a stretch to expect a human initiated action to abide by robot.txt.

Also, once you click on a link in chrome it's pretty much all robot parsed and rendered from there as well..

At bottom, all robot actions are human initiated.

Exactly, so they should ignore robots.txt !

I would say robots.txt is meant to filter access for interactions initiated by an automated process (ie automatic crawling). Since the interaction to request a site with a language model is manual (a human request) it doesn't make sense to me that it is used to block that request.

If you want to block information you provide from going through ClosedAI servers, block their IPs instead of using robots.txt.

If my web browser's extension "visits" the site and dumps it into ChatGPT for me to read its summarization of the site, what has been gained by the website operator?

Added friction. That is all a website owner can hope to achieve.

Nothing.

This is why these things - search engines, AI crawlers, even adblock and video downloaders - exist in a slightly adversarial/parasitic relationship with the sites that provide their content to which they provide nothing back (or negative, if you cost them a page load without incurring an ad view).

I use adblock all the time but I'm very aware that it can only succeed as long as it doesn't win.

Reply on news.ycombinator.com