To explain the reasoning for this proposal, by way of an example: I recently released FastHTML, a small library for creating hypermedia applications, and by far the most common concern I've received from potential users is that language models aren't able to help use it, since it was created after the knowledge cutoff of current models.
IDEs like Cursor let you add docs to the model context, which is a great solution to this issue -- except what docs should you add? The idea is that if you, as a site creator, want to make it easier for systems like Cursor to use your docs, then you can provide a small text file linking to the AI-friendly documentation you think is most likely to be helpful in the context window.
Of course, these systems already are perfectly capable of doing their own automated scraping, but the results aren't that great. They don't really know what's needed to be in context to get the key foundational information, and some of that information might be on external sites anyway. I've found I get dramatically better results by carefully curating the context for my prompts for each system I use, and it seems like a waste of time for everyone to redo the same work of this curation, rather than the site owner doing it once for every visitor that needs it. I've also found this very useful with Claude Projects.
llms.txt isn't really designed to help with scraping; it's designed to help end-users use the information on web sites with the help of AI, for web-site owners interested in doing that. It's orthogonal to robots.txt, which is used to let bots know what they may and may not access.
(If folks feel like this proposal is helpful, then it might be worth registering with /.well-known/. Since the RFC for that says "Applications that wish to mint new well-known URIs MUST register them", and I don't even know if people are interested in this, it felt a bit soon to be registering it now.)