Personally, I like the LLMs.txt standard. I see AI scrapers hitting a few of my sites all the time, and I honestly am currently ok with my content being scraped (though I still retain my rights over my content, and this does not serve as permission to steal my content should I decide I am not okay with the theft of my content). Anything that can, sometimes quite dramatically, lower the bandwidth and cost of scrapers for both myself and the scrapers sounds like an ultimate good to me. I can understand the... I guess it's a repulsion to it, since you're putting work into something that ultimately leads to your content being stolen. However, we as developers must come to understand that in the current year, developers use and sometimes rely on large language models. I myself use Supermaven and have experimented with various LLM platforms, as well as self-hosted some models. LLMs are a great tool if you can use them correctly, though I am not some AI evangelist, not by any means. I believe LLMs.txt offers a significant benefit to users, operators, and AI providers.
... And then it removed it, prompting speculation that it was a mistake.