An empirical study of harness design for coding agents
arxiv.org
arxiv.org
Basically if a Car A is performing better (be it speed, milage or in general sense) than Car B, then it is not necessarily because its engine. It could be because of better tires, better gearbox, lighter body, better usability of features, etc.
You can implement an AI feature (like AI for BI) in different ways even with the same model - via ReAct-loop, or plan-and-execute, or hybrid. You can make it stateless, stateful, RAG-based, etc. depending upon whether you want to prioritize result accuracy or depth of analysis. You can use LLM to generate either intent (requires lesser reasoning) or the queries itself (requires much more capable model).
Your harness can adapt to the underlying model's native capabilities, or can make up for its absence, e.g. query generation in above example requires your model to have MOE capabilities but intent generation wouldn't.
Cars performance is hugely dependent on use case and overall setup (suspension, engine (NA, turbo, supercharger, etc), coef of drag, etc).
Maps well onto the LLM surface.
But if you just want to improve lap times it's way less of a priority than brakes, tires, suspension setup, cooling to keep everything happy, etc.
Kind of like if your RAM has to constantly page, you would see much more improvement from upgrading your RAM than overclocking your CPU?
Because I feel that the technology and space is - so - hyped and fast moving that a lot of cultish feeling rituals seem to pop up, none of which are backed by evidence. Anthropic openly recommend giving the agents.md file an architectural overview of the code, and the one time this was studied they found the opposite - that the agents.md file is best for concrete commands about how to build stuff and such, and - not - huge overviews. This was, and still is, the official recommendation from Anthropic as far as I can tell.
And then there are the benchmarks, how feel vague and not concrete, and everyone kind of knows they're not the best cuz you can't just assign these tools one fixed number ( for multiple reasons ), but everyone still looks at them and compares them.
People share skills and superpowers and plugins and mcps and very, very few of them have and kind of proof they do much at all.
It all feels a bit weird to me, and I've been on the lookout for exactly these kinds of studies more lately, because I think having this research, even if not done on the exact newest models or not the exact, newest thing, are still - vastly - superior to the alternative.
At least 3 times last month I was asked to review a change in a .md file used by agents. And I am like: "yeah I guess it makes sense?"
It feels we need to write unit tests for this stuff, but even how to do so in reasonable time and complexity seems difficult.
It is way more than 100 lines. Why keep advertising something that is no longer the case?
190 lines with comments included so close enough
Which is the actual agent? Everything else is just glue code to connect to different LLM providers, etc
[1] https://www.induction.ai/docs/context-management [2] https://github.com/ArtificialAnalysis/Stirrup
I would expect the agent loop and system prompt to be basically the same. Is it the precise semantics of the tools (and how closely they match what a particular agent was trained on) or something else?
> Planning improves success at additional cost for weaker models but mainly reduces cost, with small decreases in success rate, for stronger models.
> Predefined tools raise success rates for models with weak bash control, whereas bash-only yields higher success at lower cost for bash-capable models, most clearly on shell-centric task types.
> context management extends execution trajectories without substantially altering agent behavior and is most beneficial under tight context budgets
> planning sustains the trajectories of models that abandon tasks too early and trims repeated verification in models that verify too long
> structured tools support models with limited shell proficiency, while bash-only enables capable models to combine multiple code modifications in a single tool call
Seems fairly intuitive to me, based on feeling. But also fairly kind of obvious; bash-only tooling has higher success for bash-capable models, compared to using predefined tools for models that aren't good at bash? Yeah... They all seem a bit "duh" to me. The final piece of the conclusion is agreeable regardless of how they arrived at it though:
> Harness design is thus a conditional systems problem in which each component should be selected for the target model, task type, and resource budget rather than adopted as a default.
I think lots of people treat the harness/model/prompts combo as interchangeable, but in my experience the quality and efficiently depends heavily on the combo of the harness/model, and using the harness + model made by the same lab, has vastly better experience compared to more "general purpose" (for the lack of a better term) harnesses. Most likely because they use their own traces when training future model iterations.
"Duh, of course Mars has canals."
Testing the "obvious", "duh" things is incredibly valuable science. It provides a more solid foundation on which to build because it reduces the assumption space.
not really, there was a recent benchmark with claude and codex and it showed no difference in ability with a harness like pi agent compared to their native harnesses, pi was in fact cheaper per task.
No. The conclusion is that:
bash-capable models + bash-only tools > bash-capable models + predefined tools
In other words, MCP was just a bunch of bullshit that maybe helped a little bit until the models got good at bash, and now it's basically useless.
The design of pipes and the philosophy of simple composable tools tied together with the Unix shell has proven remarkably difficult to improve upon.
At one point the web was bash scripts glued together. Now everything is brought into a runtime. Then brought into virtualized containers to isolate and hide from every other aspect of the operating system.
Eventually the agents will be on a runtime as well. The existing ones just aren't good enough. Rather than forking like made as a way of doing everything, exfiltration into arbitrary executables will be something more tightly controlled.
Organizations need better in-product control and auditing of the operations of the agent. They need it to work across OSes and not depend on the state of the machine, and not conflict with what else the developer is trying to do with it.
> In other words, MCP was just a bunch of bullshit
If you're just using bash scripts, where are you putting your enterprise secrets for external systems? How do you cleanly revoke them when a developer leaves your team?
MCP moves execution into a remote environment where it is easy for enterprises to secure access to internal and external systems. OAuth based access makes it easy to audit and revoke tokens. Central HTTP interface makes it trivially easy to monitor and audit.
They solve different problems.
What is a noticeable improvement with something that struggles to read a message longer than 200 characters without missing information in the middle, may be a 0.000000001% improvement with a model that... almost never misses info in the first place.
I think until we have actual repeated-use measurements tracked over time (eg consistent prompts used to do the same tasks, count number of hallucinations and errors and bugs over a long period of time) you won't really have any idea of which model is better
I also think of it like a car - some just feel better to drive even if they are materially worse in other measures. until you start measuring the metrics important to you (eg MPG and cost of maintenance over a long period), you have no idea which car is actually better suited for you. and the fact that you can only do so with a limited number of cars (or hours available to work, or money to burn on tokens) means there's no true measure approaching objectivity
comparing x10 to x100 doesn’t necessary inform you about x100_000 to x1_000_000
So i can skip this study.
And there might be a point to these arguments, vaguely. However:
There never seems to be - any - kind of counter example or reasoning behind the rationale. You have an in depth and empirical study, done by researchers who, frankly, now their shit (most of the time)
And on the other hand a random internet comment saying "nope" because...the models aren't the latest.
If the latest models really would make a difference, you should at least provide some kind of evidence towards that. As it stands though, every time these comments come up this is missing.
There seems to just be a vaguely defined understanding that "everything changes all the time, and nothing you ever research is transferable to state-of-the-art models"
Which brings me to my second point about these kinds of arguments:
LLM models often - aren't - fundamentally different. Yes, they are vastly more capable. And yes, there are emergent properties. But at their core, they function very much similarly. And for quite a while now, there have not been any of these drastic changes we saw when LLMs first become "good enough" for agentic coding.
I am tired of dismissing empirical evidence and studies every. single. time for reasons without evidence and seemingly a vague sense of "no, but my model is different"
This is a field that changes significantly every few months, so using 1 year old models essentially invalidates the entire report, as they are 2 model generations behind and the newer generation models are heavily RL-d around their harness. For example they put a lot of emphasis on context management but that kind of context management isnt done anymore because many models now have 250k or 1M context and inferencing the models heavily rewards cache hits so you should never touch the context until you decide to compact. They comment on plan mode, which is something that Claude decided to remove from their client because they dont consider it needed at all anymore.
Anthropic appears to agree frontier models don't need in-session planning tools.
https://rahulmax.com/notes/how-i-keep-the-ai-bill-down/
In their case, context management pays off more the tighter your window. Their gap between managing and not managing is 35.7 points of success rate at 32k and 2.7 points at 128k. My version of that was a rule I stick to, as much as I can. I checkpoint a session at about 25-30% of the window, write the state out to a PROGRESS.md and a JSON file of the requirements, and start fresh. This restart costs me 30 seconds, since a bloated session doesn't get any cheaper the longer you stay in it.
Also worth knowing that the models are Nemotron-3 and Mistral-Medium, not the frontier models most people here are paying for.
Lately, I've been finding that the game of telephone of handoffs causes more mistakes than just letting the session run longer. Not too mention the wasted time waiting for agents to poke around as they bootstrap a new session from the handoff.
Of course, I still have handoffs for the large-scale plan being accomplished, but I've been having better results letting an agent finishing what it starts. The game of telephone is a painful waste of time when it goes wrong, which is far too often for me.
Though of course every single element of their application's actual UX is pushing hard in the opposite direction. The cynical part of me can't help but notice that there's a bit of a conundrum here: using coding agents more effectively seems to involve working pretty hard at giving the agent vendor less money.
I would have to imagine, that Luna would very much fall into the camp of "bash capable". At which point- it seems to me that adding any tools beyond just Bash requires some rigorous testing and verification that value is being added.
The research seems to agree with you, though. The paper calls out that for “bash capable” models, adding tools to do things bash can already do doesn’t improve performance.
Vaguely the same result as RAG. Unless you’re in specific domains, you won’t beat handing the agent a shell and grep.
This has been my conclusion as well, and I'm doing my best to try and back this up quantitatively. In a perfect world, I could smite all of the internal MCP servers in my Corp environment and replace them with REST/GraphQL. No one using any of these servers is hitting them with models that would perform worse orchestrating with bash- and some of these folks are running harnesses with awful MCP clients.
Is it the harness that enables the LLM to use tools and implement plans, or is it the short leash that, through many guardrails, ensures that the LLM follows the desired path?
I see both meanings used. The former is still plausible, but I see many people using the second interpretation.
Always looking for more harness evals, although I'm going broke running them across all these models and tasks.
I tried to get my lawyer-mom switched to Linux a bit ago, and she loved it in generally, but none of the Office competitors had good enough compat to work.
The only other thing keeping her on Windows is Adobe PDF pro, since it can do OCR where, when you edit it, it reflow the text in a font that matches the scanned in one to look like the original. (I also got weird "This feels like it enables fraud vibes" from this, but, no, turns out it's a totally common workflow for lawyers to need to do this ... I hate it.)
Cool to see 1/2 of the problems keeping her on Windows solved.
Don't worry about that - all the agents/harnesses/tools that produce .doc or .docx files do so with LibreOffice now. In a short time people would rather tell their agent "produce this file" and get one that is 99.9% compatible with Word than actually open Word and use the builtin copilot.