Bluesky / ATProto has a proposal for User Intents for data. More semantics than robots.txt, but equally unenforceable. Usage with AI is one of the intents to be signaled by users

https://github.com/bluesky-social/proposals/tree/main/0008-u...

Small difference. Its called llms.txt

https://llmstxt.org/

If they don't comply with robots.txt, why would they comply with anything else?