That’s so interesting! I know a few of the individuals and really am curious on how they interact/work together. Cool stuff
13 karma · joined March 10, 2026
Official OpenAI documentation: https://platform.openai.com/docs/gptbot
On the broader point, I hear you, but I think there's a middle ground. Not all content is public knowledge. Some of it is premium, proprietary, or behind a paywall. The people publishing it should get to decide whether it becomes free training data.
On the AI docs concern, fair point. To answer directly: I've confirmed the obfuscation defeats any scraper reading raw HTML via HTTP requests. Whether GPTBot or ClaudeBot use headless browsers internally, I honestly don't know. The README threat model lists headless browsers under "what it does NOT stop" for that reason.
Happy to have a11y experts poke at it and point out gaps.