How are you measuring Claude Code and Codex performance? | Hacker News Reader

How are you measuring Claude Code and Codex performance?

3 pointsachalpandey9 comments

I think coding benchmark results don't represent our messy reality. As they

1) Use purpose-built test harnesses We use Claude Code or Codex

2) Test one-shot tasks We work in sessions

Sessions are messy, we start with a large primary task, then some cleanup, an adjacent fix here another over there. We start, stop, and change our minds.

That changes both cost and quality. Cache TTLs expire. Context grows.

I am working on creating one. Here's my rough plan: 1) Use Claude Code and Codex 2) Use session shaped workloads. Stitch multiple SWE bench verified tasks into one big session. 2.a) Use tasks from the same repo to ensure topical continuity. 3) Headline metrics: dollar cost vs quality. 3.a) Secondary metrics: turn count, time to completion

Open Questions a) Does this problem make sense? b) Does my benchmark spec make sense? c) Are 10 SWE bench verified task starting with a hard one the right workload shape?

I've actually been working on a solution for this problem! https://www.stet.sh/

At a high level, it

- Mines tasks from your merged PRs/commits - Replays them in Docker containers with different harness settings (change model / reasoning effort / AGENTS.md / etc) - Grades the patches on various attributes (tests, equivalence with human patch, code quality)

The goal is to get a sense of how agents perform on your tasks, with your context, using the tools you do.

This is currently one-shot but I'd definitely like to explore session-based benchmarks as well. There are some interesting papers that just came out on this https://arxiv.org/abs/2606.29957 https://arxiv.org/abs/2606.30573

Thank you! Both of those papers are super new and super relevant.

The only big gap left is that they aren't using claude code/codex as harnesses. I'll try to reuse their constructed user sessions.

PS: Your work at stet is also interesting. That's definitely a problem right now that's hard to track. The only real solution is more robust CI/CD. I have since added harder validation like essentially running a full benchmark run on every prod push.

UPDATE: turns out "some" models know how to game the premise. They simply lookup the solution to the exact solved SWE bench problems! Haha

Because I don't want to impose artificial constraints like no network access, I'm going to try the other two harnesses from the paper shared by bisonbear. Primary thing I will extend is to setup the run such that it uses claude code/codex instead of custom test harnesses

as a tip - models will always find a way to cheat, you will probably need to impose some restrictions on what they do / are able to access in the sandbox environment

see https://cursor.com/blog/reward-hacking-coding-benchmarks

Hmm, I am trying to benchmark cost/quality for real world sessions. In that scenario "model resourcefulness" and efficiency is actually a good thing.

Why spend tokens working through a solution when you can simply look it up?

Thoughts?

Thank you for sharing this blog it's a good read! You are definitely plugged in to the benchmark space! :)

It depends on what you're measuring. I agree that model resourcefulness is useful, but if you're trying to simulate real user sessions, then Claude looking at upstream Git and fetching the answer directly is somewhat worthless.

In my case, I'm trying to measure how coding agents perform under realistic scenarios when implementing tasks, as a proxy for how agents perform when used by actual users for those same tasks, so it's important to ensure the agents are behaving realistically instead of "cheating" and looking up answers.

Happy to share resources! I've been pretty deep in the space :)

Looking for feedback and thoughts. Here's a link to my one-page spec: https://docs.google.com/document/d/e/2PACX-1vRu5Fv5-KTJDnCEx...

Thanks for sharing this. For session-shaped benchmarks, how would you keep the evaluation fair when cache state and accumulated context differ across Claude Code and Codex runs?

Here's my current plan, the "session" will be made up of multiple SWE bench tasks stitched together.

Each "task" is the equivalent of a new user query and we also pre-program "cache expiration" (sleep for 5 mins) into the session. This ensures parity across providers (both default to 5 min TTLs).

The goal of this exercise is to tease out how Claude Code and Codex differ in managing their context and how that impacts cost and quality for the same simulated session.

Reply on news.ycombinator.com