Hacker News Reader

Comment by lieret | Hacker News Reader

Cool study, we definitely need more principled studies on the role of harnesses. I'd also say that there aren't too many benchmarks where the more complicated harnesses consistently outperform extremely simple agents. But I'm also biased, because I wrote https://github.com/swe-agent/mini-swe-agent/ , which is probably the most minimal agent out there (it started as just 100 lines, all included), and it's used in a lot of benchmarks like DeepSWE, terminalbench, programbench (seems like it's still top of the ranking for TB3, but wasn't evaluated with the best models on TB4).

View on HN

> Minimal: Just some 100 lines of python for the agent class (and a bit more for the environment, model, and run script) — no fancy dependencies!

It is way more than 100 lines. Why keep advertising something that is no longer the case?

I think they are referring to the file https://github.com/SWE-agent/mini-swe-agent/blob/main/src/mi...

190 lines with comments included so close enough

Which is the actual agent? Everything else is just glue code to connect to different LLM providers, etc

190 is not close enough. It is close to double.

Minimal agents also allow room for more focused add-on tools/infra. I'm working on a context management layer[1] and it's very difficult to do well. In our benchmarking, we've observed simple harnesses like Stirrup[2] outperforming more elaborate ones.

[1] https://www.induction.ai/docs/context-management [2] https://github.com/ArtificialAnalysis/Stirrup

I'd really love to see more studies about effectiveness of AI in general. As in, what works best and how to use it and such.

Because I feel that the technology and space is - so - hyped and fast moving that a lot of cultish feeling rituals seem to pop up, none of which are backed by evidence. Anthropic openly recommend giving the agents.md file an architectural overview of the code, and the one time this was studied they found the opposite - that the agents.md file is best for concrete commands about how to build stuff and such, and - not - huge overviews. This was, and still is, the official recommendation from Anthropic as far as I can tell.

And then there are the benchmarks, how feel vague and not concrete, and everyone kind of knows they're not the best cuz you can't just assign these tools one fixed number ( for multiple reasons ), but everyone still looks at them and compares them.

People share skills and superpowers and plugins and mcps and very, very few of them have and kind of proof they do much at all.

It all feels a bit weird to me, and I've been on the lookout for exactly these kinds of studies more lately, because I think having this research, even if not done on the exact newest models or not the exact, newest thing, are still - vastly - superior to the alternative.

> People share skills and superpowers and plugins and mcps and very, very few of them have and kind of proof they do much at all.

At least 3 times last month I was asked to review a change in a .md file used by agents. And I am like: "yeah I guess it makes sense?"

It feels we need to write unit tests for this stuff, but even how to do so in reasonable time and complexity seems difficult.

I think that generally speaking, the tighter the feedback loop, the better AI performs. Ask it to do things that are fast and easy to check and I suspect not much else matters. That's why AI's eating programming but has mitigated success elsewhere, feedback loops in the digital world are orders of magnitude faster.

The various “minimal” agents (at least Pi, mini SWE, dsh minimal) seem to benchmark quite differently and none is clearly better in all cases. Do you have any thoughts about why?

I would expect the agent loop and system prompt to be basically the same. Is it the precise semantics of the tools (and how closely they match what a particular agent was trained on) or something else?

Reply on news.ycombinator.com