agentsmd is a ridiculous concept, it feels like a huge psyop that theres so much hullabaloo about this ridiculous markdown file.
Most of the agents.md and what people use it for / write into is does, in fact, not make a difference.
Now, sure, this study is a bit old for LLM standards - as everything beyond the current month is - but
a) I haven't seen any tangible evidence to the contrary and
b) Since the basic inner workings of LLMs haven't changed I'd be sceptical of this not still applying.
I think one major side effect of LLMs moving so fast is that best practices and how to use this tool is very much not catching up as fast.
No one knows what is best and what actually makes a difference, doubly so because LLMs are / very / hard to quantify - even benchmarks themselves are very rough estimations.
People do, in fact, use stuff which makes no difference all the times.
> We conclude that while context files are useful for specifying non-standard coding practices, any attempts to improve performance should be rigorously evaluated before deployment.
The purpose of AGENTS.md is not to improve "coding performance" as the study looked at, it's to give an agent practical instructions that are useful to your specific workflow. For example you want it to use a certain format or specific tools for your project. This is stuff that can't be learned during training and must be loaded into the agent's context at the project level.
I suspect eventually this will move to fine-tuning, there are multi-tenant LoRAs on the horizon that will let us keep paying per-token on FTs
They also recommend starting with an /init command, which is also something the study very specifically called out as "having a marginal negative effect"
And from personal experience, I can only confirm that many people seem to see this as the main purpose of agents/claude md files - a persistent architectural overview of your project.
Quick Edit: My point simply being that I think it's understandable if some people don't understand the big deal about these files because they've had a drastically different experience than other people - anthropic themselves recommend apparently totally ineffective practices on their website, and the starter tool present in many harnesses seems to even have a (marginal) negative effect.
If it's useful to an LLM its useful to a human, information should be treated as human first, the LLM will come along with it.