Comment by vidarh | Hacker News Reader

Comment by vidarh | Hacker News Reader

Testing it against Kimi is potentially skewing the numbers massively. Kimi has a number of quirks that requires behaviours that e.g. Claude or GPT doesn't.

Harnesses that are built around needing to work with "weird" models will need to deal with that, such as Kimi's tendency to get stuck in tool-call loops.

Harnesses built to deal with e.g. Anthropic's models primarily, do not need to deal with that.

Claiming on the blog that this gives Kimi Code no home field advantage seems like a dicey assumption. I haven't dug into the newest Kimi Code much, but the older Kimi CLI included several tools that were clearly specifically aimed at working around that behaviour - when I copied their checkpoints and "dmail" mechanism into my own harness, the performance with Kimi improved dramatically, but it made zero difference against Anthropic models.

That doesn't make the data worthless - it's clear you shouldn't use Clade Code to work against Kimi. But it does significantly limit the utility of it.

Good points! That’s a known limitation of our v1.0 benchmark with Claude Code. For v1.1, we're expanding both harnesses and models to evaluate the full harness × model matrix. This will highlight interaction effects (showing which harnesses suit which models best) and let us publish a full compatibility matrix. We shared more details on our roadmap in our blog post (https://runta.com/blog/introducing-frontierharness-eval/).

That's great. It'd definitively be very interesting. Not least as an indicator of which harnesses to dig into the code of.

Kimi-cli is actually very interesting in that respect for the checkpoint / messaging mechanism I mentioned; it's basically almost like a model-initiated partial compaction of the end of a conversation to prune investigations that happened on the main agent loop - it'll be really interesting to see if it helps on any other models.

So oe thing that'd be really interesting to see when you expand to other models, would be if you mine the traces from those harnesses and generate stats on which tools get called. Seeing which models manage to take advantage of custom tools from which harnesses would be quite useful.

Afaik the reverse is also true. Some models are trained for specific harness setups so work better/more efficiently in that harness. If you give a model trained on a Read/Write/Bash tool setup a harness with only Bash where it needs to generate shell commands, I'd expect it to perform differently.

Related, there was a small debacle when Claude Code yeeted out a prompt change telling Opus to use Bash for everything. It was speculated it was an optimization to try to make it more efficient by encouraging it to chain a bunch of "tool use" commands into a giant shell command. However, they didn't communicate it clearly and it broke a bunch of setups that expected/hooked tool calls

There's also the (anecdotal?) tradeoff that dumb models tend to do better with smart tools and smart models tend to do better with dumb tools (smart tools -> easier to use but require more context usage for specs, dumb tools -> harder to use but more versatile and save context)

Reply on news.ycombinator.com