HarnessTax: How Much Does the Harness Matter for Coding Agents?
harnesstax.github.io
harnesstax.github.io
I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token count?
My experience is that the harness is mainly polish preventing failed tool calls, bad edits, stuff like that, but doesn't make much difference to the overall "intelligence". But that opencode seems slightly more robust against stupid errors than out of the box Pi due to the additional context it forces through every thread.
The harness becomes more and more important, the smaller the model is as you need to offload context management as well as memory to the harness. The big models basically just need a bash prompt tooling and you let the model manage everything inside its own context.
disclaimer, I use opencode and have customized parts of it, and will do more, but it is a solid foundation and comes with more out of the box than pi
pi is too minimal for me, I'd go back to my custom built harness if I wanted to be back at that level
Such as?
They also have instructions about how to format certain output, which conflicts with the instructions we have in repo. I only discovered yesterday because we were wondering why the agent kept picking certain tools.
At this point when all the models have been trained on all available data with the similar algorithm,
1. either you get more data which is not feasible,
2. or get a better algorithm - a possibility ,
3. or write a more targeted harness.
Harnesses for legal, medicine and all are the ones which are getting focus for this reason. Writing benchmarks for these targeted harnesses would be a catching task
I don't think yet that a general benchmark will capture what is needed, but a gain/loss of function along with improvement / loss along the ability of that function seems to be a good path forward.
Personally jumping around a lot to get a feeling for exactly that, and these days liking the Grok harness out of all of them the most
They used to have one chart with {model X harness} for a subset of combos, looks like that is getting an upgrade
- Most people use a harness because of its subscription (most companies pay Anthropic) - All model benchmarks are biased and gamed, harness benchmarks are too few to matter - Everyone is just guessing, acting on sample sizes of 1 and trust me bro vibes
E.g. for editing files with Claude models you should use Edit(file_path, old_string, new_string, replace_all) but with GPT models you should use apply_patch_call(patch) (where patch is a custom patch string with custom grammar).
It appears newer models are better at narive harness tool calls and worse at custom tools that look similar to default tools.
https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/
I can understand this take 4-5 years ago but I have no idea how that's your position in 2026
not RegEx ... more like a "fuzzy" DSL.
Or is that just why my model likes to break out of the sandbox, going strait to exec shell command and editing files using python on the cli?
This is true. For example, the other day I was doing some testing with DeepSeek-V4-Flash-0731, and I gave it a tool called "EditFile" with "old_content" argument. It ignored the tool definition and called it with "old_string" instead, presumably because it was trained on a bunch of Claude Code traces.
So either make sure your tools look nothing like the tools in the popular harnesses, or make them similar.
identical. similar is what you tried and it didn't work.
We are still witnessing the same underlying problems of transformers.
[0] Think back to all the publicity stunts like the Hugging Face. They are meant to convince you that the agents have somehow progressed past the transformer limitations when those publicity stunts are actually expressions of transformer limitations.
The burden of proof that this isn't just a publicity stunt again is squarely on them.
I’ve seen this before and the conclusion ends up being “use the native harness”, which I don’t think is an acceptable solution.
const usePatch = model.modelID.includes("gpt-") && !model.modelID.includes("oss") && !model.modelID.includes("gpt-4")
Pi uses its own tools, like Armin wrote in the linked article.
Having a coding harness is critical but the differences between them are overstated.
Personally, I’ve replaced OpenCode with a thin wrapper around Pydantic-AI as the pythonic analogue to Pi-Agent for headless use via Hermes
They’d all do the job - I just prefer to compartmentalize for access control.
Keeping the harness’ surface area tiny had the added benefit of preserving my understanding and being able to adapt it to my preferred workflow effortlessly
It's the killer feature from me personally, often when wanting to troubleshoot for example things like Kubernetes workloads. LLMs are now really good at it, but we doesn't want them to like delete a pod.
Other harnesses like Codex have often on static rules, like the allow/deny of claude code, that can filter out based on regex. It's quite good already, but sometimes the model can find a way to write something that wasn't anticipated, or in a convoluted way.
After, I guess it's something that can be added in an open-source harness like Pi, and add like this new Jev model or something else equivalent
Personally, I’m not convinced that a cascade of LLM judges lets me sleep soundly at night.
My agents stay in their sandbox and escalating changes to that env or prod is solely in the (human) operator’s domain.
Obviously this is less convenient but I think it’s a necessary tradeoff for security and correctness.
That's really interesting. I like Pydantic AI a lot and wondered why all of the harnesses seem to be written in Javascript instead of it. What do you use it for headless, though? I haven't tried Hermes or similar yet, so don't have a handle on what you do with them.
Turns out I didn't need a fancy TUI at all when desktop, web and mobile interfaces are already provided by or available for Hermes.
Optimal harnesses use concurrent execution + subagents and are not stuck on one model. Cost and performance are impacted GREATLY by these tactics, regardless of the native agent context (instruction). This kind of single-harness analysis is shallow and misleading, although the finding that "Provider-specific optimization does not guarantee the best pairing" is probably correct, depending on how you measure.
It is a starting point.
Ensuring good test coverage and quality is how you purchase trust in the work agents do. This also reduces the context problem - a test collection has no recall issues, it just runs every time you call it, the whole battery, checks all the things we could check by code in one fast tool call. For the rest, the things we can't test by code, I use manual testing.
A large class of problems are intent divergence, when the model passes tests but it didn't do what I asked. For that I keep a log of all user messages in the project history and review it with agents. This intent alignment is repeated from time to time to catch drift.
So I see the "why should I trust the work agent did?" problem as a combination of 1. ensure good testing 2. review intent alignment.
In my experience if you're using OpenAI/Claude models and paying API costs, almost every other harness beats Claude Code/Codex in cost.
I wonder if the same is true for the smaller models in the 9-32B range? I would expect that these models need more steering, but again I was not expecting this result either.
With coding agents, what I’ve noticed is that a simple task can often be handled with a fairly simple harness. But the hidden cost is really around context. One of the more interesting things I’ve seen is that two different harnesses can make a similar number of model calls while consuming a very different amount of context.
I think I recently came across a paper comparing Claude Code and Pi that touched on this. More context, more tooling, focused context, simpler loops, all of these can lead to very different costs and performance, even when the number of model calls looks similar.
That said, system prompt (which to your point also comes from the human) matters a lot too. For example a coding harness that doesn’t a have a plan mode will keep trying to write underbaked ideas to code instead of working with the user on the plan. That said, a “system prompt” can also be the first prompt you send to the agent in a vanilla chat window, so I do like your point about the user being the real harness
Do harnesses and therefore sending the queries directly to the LLM providers have caching and other benefits that OpenRouter does not provide? Would I get any of those benefits if I simply proxied any requests to the major providers' harnesses through OpenRouter? Or only if the requests go straight from the harness to the provider's API?
As per your response it sounds like at least caching would happen for any provider regardless of the request's origin.
Its literally me realizing they don't have a good solution for it. Context should be local. It saves money and potential LLM confusion.
Plus i can foresee a bunch of other deterministic context and session handling being just plain unrealized due to the default we have currently
ZCode is great. But as tied to GLM as CC is to Claude models (i.e. you can change models but it will cost you UX, performance, money. OpenCode works pretty well for all models, but for CC and Z.ai their proprietary harness is just a little better.
On the fence now what i will continue using. Probs ZCode will eventually give me the same frustration that CC gave me. so OC may be best.
Though for cc you are correct, using it with your sub draws from extra usage.
Every 3rd party open-source harness I know of supports ChatGPT subscriptions.
It isn’t explicitly allowed in OAI’s TOS, however they publicly support Pi and OpenCode’s usage of their Oauth, and because Codex is open-source, it means the machinery to support Oauth login is open-source under Apache 2.0
- Prevent subscription usage on third party harnesses
- Append "Co-authored by Claude ..." to commit messages
Anthropic and Google don't let you do it (there maybe be few more).
I want one place where I define project prompt, one place where I define project agents, MCPs, config. And all of this immediately rules out any lab-provided harness.
oh-my-pi then matched most closely to what I had built. I've been running it for the past 2 months with no complaints.
Claude implementation is technically ToS breaking, but my 3 accounts are fine so far.
This is interesting. We built our own harness (CodeBot, an agent for Delphi) and it currently uses OpenAI models; we tuned it for specific behaviours and patterns and I find its behaviour better than Codex. Same models underneath.
In the case of DeepSeek-V4-Flash-0731 on deepinfra.com there was little difference when both used high reasoning. In the case of that same model on together.ai there was a substantial difference between the two (high reasoning for both again). When using together.ai with Kilo Code the LLM was having a lot of trouble making successful edits. In some cases that meant a lot tries at using the tools and in others it worked around by running scripts. Meanwhile it used the tools from my harness just fine. I've specifically tried to make my tools easy for all of the open weight LLMs to use correctly. That was inspired by getting some errors from Kilo Code at the beginning of the year telling me that the model was having trouble and I should use a smarter model.
And they also tune the inference of the models behind it.
You're using different tooling every day. Hard to benchmark.
Compare this to the Codex CLI that is no fluff, minimal and just does it's job. Open AI models are in a totally different league to Anthropic (in the real world) so no loss really.
"Why pay the waste disposal tax? Dumping into the ocean is free!"
Pi actively omits any sort of guardrails and sandboxing in the name of speed and simplicity, so it is not shocking that it is faster and simpler.
Doubling the cost of something in the name of vague security is standard operating procedure for big enterprises, maybe even quite cheap.
(I do think Pi should ship with more than zero builtin sandboxing though)
Funny to me how stuff is described these days. Seems like some people lack the full understanding of all these different systems and really overcomplicate their harness setups or use super bloated all in one tooling.
I'm looking at changing my AI harness and am definitely interested in this
They add a --tools flag, which can only add tools onto the built-in tools, so if you wanted to sandbox pi.dev by adding sandboxed versions of the existing tools you will have to run pi.dev via
pi --no-tools --tools tool1, tool2, tool3
So if you want to sandbox your agent, guess what you'll have to do? Yep, you have to supply --no-extensions.
Here is how to run pi.dev under its most minimal configuration under a sandbox:
pi --no-tools --no-extensions --tools tool1, tool2, tool3 -e ./your-sandboxing-extension
Um, no? Have you actually read Claude Code's system prompt and tools?
Here's Fable's system prompt: https://github.com/navanchauhan/agent-autopsy/blob/6d9c00e54...
There are a few lines of what I would call "for security and alignment purposes", but it's not the majority. And the majority of the bloat is from the tool descriptions.
The system prompt in this case has pretty much nothing to do with the built-in guard rails, that is: you cannot sidestep Claude's guard rails by editing/replacing the system prompt (which you officially can with Claude Code's `--system-prompt-file`).