> The robots.txt is the real story.

Ironically that reads to me like a typical claudeism