That’s the whole point. The site owner doesn’t want their information included in ChatGPT—they want you going to their website to view it instead.
It’s functioning exactly as designed.
It’s functioning exactly as designed.
Also, once you click on a link in chrome it's pretty much all robot parsed and rendered from there as well..
If you want to block information you provide from going through ClosedAI servers, block their IPs instead of using robots.txt.
This is why these things - search engines, AI crawlers, even adblock and video downloaders - exist in a slightly adversarial/parasitic relationship with the sites that provide their content to which they provide nothing back (or negative, if you cost them a page load without incurring an ad view).
I use adblock all the time but I'm very aware that it can only succeed as long as it doesn't win.