bisonbear

I've found that building a benchmark is something that's easy-ish on the surface level, but has many layers of technical complexity the deep you go down the rabbit hole.

For example, a tangent on grading. Do you use test passing and/or LLM graders? If you're using an LLM, it's fairly easy to spin up a basic LLM grader. But - is that grader calibrated? Is it biasing towards same-model agents? How many results do you need to be above the noise floor? What is it even measuring, and is that aligned with what you care about? Does it have the right information to make those judgement calls?

I'm super curious how what you've been using this workflow for? Also happy to answer any questions / discuss further!

They compare it to 5.6 Terra, however https://cognition.com/frontiercode puts Terra at about 1/2 the price

Also have to compare to the recent Grok 4.6 release, which appears to straight up be better AND cheaper

Hard to understand why anyone would choose 3.7 Flash under these conditions.. is Deepmind still a frontier lab?

Also have to compare to the recent Grok 4.6 release, which appears to straight up be better AND cheaper

Hard to understand why anyone would choose 3.7 Flash under these conditions.. is Deepmind still a frontier lab?

However, as a self-admitted eval true believer, I remain bullish that publicly available eval capabilities will catch up to current model intelligence, and start to be able to capture more and more of the subtle components of the agent experience, such as interactivity and code maintainability. Meta and Scale have some evals in the interactivity space that seem pretty promising https://arxiv.org/abs/2606.29957 https://labs.scale.com/papers/swe-interact.

Human usability of the agent should definitely remain the top priority, but IMO we focus too much on the subjective, and not enough on the measurable/objective now, and I think/hope we'll see more companies move in the direction of Databricks

Without the insight into agent performance, any changes like this feel like a gamble to save $$ at the cost of developer productivity

I'm actually working on building generic repo-specific benchmarks at https://stet.sh ;)

However what I wonder how we make these decisions on what to remove. Certainly some subset of the skills / AGENTS.md / prompt / tools do actually help agent performance, the problem is in identifying those, throwing away the rest, and validating that this didn't actually hurt the agent performance

Sharing some challenges / learnings from the journey so far:

* Tests alone are non-representative: an agent can pass tests but write the code in a way that’s subjectively worse than another test-passing result

* However, tests are still important and the best form of deterministic evaluation we have

* Creating executable environments to run the tests is challenging, especially when considering that this should be doable for arbitrary repos

* Selecting tasks that discriminate is an art - they need to be challenging, but not too challenging, represent the variety of work that’s done in the repo, and contain tests that failed prior to the change

* LLMs / agents are very powerful at judging code - but doing so in a way that is calibrated, consistent, and representative of codebase standards requires careful rubric creation and grading

* I’ve shifted towards using agents (from “static” LLM calls) to generate the rubrics, and to later grade the rubrics. This is powerful but even more non-deterministic

The first step I see towards this is building a generic, reliable, and accurate *fitness function* for codebases - turning PRs into gradeable tasks that an agent can solve, and improve on.

I’m pretty curious how others are optimizing the coding agent harness now, as this has been a huge pain point for my company (we’re pretty much relying solely on vibes).

> Verifying quality is orders of magnitude harder than "did the tests pass"

Agree that agentic grading is the future here. Cognition's Frontier Code is probably the best large public benchmark at this. You attribute agent quality issues to RLVR's binary pass/fail, however I wouldn't be surprised if labs are already supplementing that with rubrics as rewards to train more 'tasteful' models like Fable.

What can a practitioner do? I think there's promise in turning the optimization machine to the harness itself - building out a representative dataset of tasks on your repo, grading agent quality on them across various configurations, and optimizing [AGENTS.md / SKILLS.md / workflow / model / harness / tools] on that signal. High quality grading is still very hard, but it's more tractable at smaller, repo-level scale, and you can afford slower, more expensive verification for each task. You only need it to be right about your codebase's standards.

> In fact, it's not hard to imagine that if a model could reliably tell good code from bad, it might have written the good version to begin with

Pushing back slightly - detecting slop and discriminating quality is easier than generating it (why code review is so effective), and why grading is viable at repo eval scale even if it's much harder at RL scale.

Everyone is flying blind. For example, I am genuinely interested in trying HumanLayer, but would likely want some harder evidence (beyond anecdotes) that it's actually making my agents more effective before rolling out to an enterprise team.

I'm building this harness optimization loop @ https://stet.sh if curious

In my case, I'm trying to measure how coding agents perform under realistic scenarios when implementing tasks, as a proxy for how agents perform when used by actual users for those same tasks, so it's important to ensure the agents are behaving realistically instead of "cheating" and looking up answers.

Happy to share resources! I've been pretty deep in the space :)

see https://cursor.com/blog/reward-hacking-coding-benchmarks

At a high level, it

- Mines tasks from your merged PRs/commits - Replays them in Docker containers with different harness settings (change model / reasoning effort / AGENTS.md / etc) - Grades the patches on various attributes (tests, equivalence with human patch, code quality)

The goal is to get a sense of how agents perform on your tasks, with your context, using the tools you do.

This is currently one-shot but I'd definitely like to explore session-based benchmarks as well. There are some interesting papers that just came out on this https://arxiv.org/abs/2606.29957 https://arxiv.org/abs/2606.30573

Personally, I will continue seeking out high-quality music/art/movies/books that speak to me, and most of my friends do the same. There will always be a demand for human-created art, regardless of any plagiarism or replication by labs.

This is also generally where I've landed - keep the AGENTS.md super light, and link out to docs as needed. Same idea with skills as well. Basically, preserve the context window at all costs.

The part I'm curious about is, when we're making the sorts of behavior changes you're describing on shared repos, how do we actually measure and quantify impact? It's one thing to tell the team that the agent should perform better, and it's another to say that you made the agent 5% better across a variety of tasks for every dev in the repo.

This has been bothering me for a while - the entire dev community is running on vibes when talking about AI. We're operating in an old paradigm, thinking that smart and logical additions to AGENTS.md result in good agent behavior, when in fact agents behavior is such a black box, that measurement is necessary.

> Even when all the rigging is controlled. (Which implies we need multiple experiments to compare against.)

Even the rigging is hard to control - Anthropic has an interesting piece on this here https://www.anthropic.com/engineering/infrastructure-noise

I'm also noticing similar patterns with needing to update AGENTS.md / skills per model release. E.g with Opus 4.6 -> 4.7, it became much more instruction adherent, so instructions written for the prior model generation might cause unexpected behavior in the new generation. I'm also convinced that an optimal AGENTS.md for Codex is not the same file as an optimized CLAUDE.md for Claude - the model personalities and behaviors are so different that we probably need to tune the instructions differently as well.

- Keep it concise, use progressive disclosure / nested AGENTS.md for information expansion - Give agent the high level repo structure if necessary - Have a "why" section to align the agent, high level, what your code is doing - Keep behavior instructions positive where possible, eg Always clarify intent before acting

For example, if I write a bad AGENTS.md for a repo with 100 engineers actively working in it, then every agent for every engineer gets worse, without anyone really noticing.

I think we should move towards data-based tuning of AGENTS.md, testing out changes, gathering data, and then making a decision on whether or not to ship it.

To solve this, I've built an agent-native tool to run evaluations based on merged PRs in your codebase. Basically you can ask Claude to evaluate whether the skill made things better/worse on real tasks, and to then iteratively improve it

Stalking your profile (sorry..) I see you're pretty deep in the eval space, so I'm super curious what your approach has been to being rigorous for things like skill changes?

Snapshot of the results (sorry for busted format, ask your llm for dataviz. cant seem to format a good table in the comments)

Opus 4.7 on GraphQL-go-tools:

Low: 23/29 pass, 10/29 equivalent, 5/29 review-pass, custom avg 2.598, $2.50/task, 384s/task

Medium: 28/29 pass, 14/29 equivalent, 10/29 review-pass, custom avg 2.759, $3.15/task, 451s/task

High: 26/29 pass, 12/29 equivalent, 7/29 review-pass, custom avg 2.670, $5.01/task, 716s/task

Xhigh: 25/29 pass, 11/29 equivalent, 4/29 review-pass, custom avg 2.669, $6.51/task, 804s/task

Max: 27/29 pass, 13/29 equivalent, 8/29 review-pass, custom avg 2.690, $8.84/task, 997s/task

(custom avg is a set of rubrics used for llm-as-a-judge, graded out of 4)

Practically, the results indicate that medium has better outcomes, or at least the same outcomes, considering variance, as higher reasoning efforts, at a much lower cost/time.

> Progress. /s

pretty much, lmao. my theory is 4.6 started thinking less to save compute for 4.7 release. but who knows what's going on at anthropic

https://www.stet.sh/blog/opus-4-7-zod

I think this line around "context tuning" is super interesting - I see a future where, for every model release, devs go and update their CLAUDE.md / skills to adapt to new model behavior.