Hi, I'm a YC backed founder and dev building an open source harness. I was benchmarking on terminal bench 2.1 and deep swe 1.1, but it's painfully obvious how bad those benchmarks are and why they dont represent real world coding.

So I'm making a bench that you can't benchmaxx.

A few knocks on the normal benches:

    So disgustingly expensive to run

    Contaminated (trained on), or private so cant run it

    Binary results per task (this means that its hard to measure gaps in capabillity, your tasks need to cover the distance between frontier and mid models, and if it is not granular enough, it does not capture it. 60 tasks on terminal bench can be super easy, 19 can be impossible, so the difference in fable 5 and sonnet 5 is measured by the few percentage points of the 10 tasks in the middle)

    Saturated easily (once you reach a certain percentage, you need to make a new benchmark which is hard to do)

    Coarse grading (Agent can output a correct, but different implementation than expected)

    Penalizes time heavily (You want your agents to iterate in the real world, not be one and done)

This is the target benchmark spec:

    Hard to contaminate

    Hard to saturate

    Deterministic and bulletproof grading

    Continuous score

    Cheat-resistant

Jcode bench v1 is all of these. They are optimization tasks of three extremely common functions that would have real world use if optimized. The model is given some reference solution for the function, and asked to optimize it. Because there is only three tasks, it is cheaper to run. Every time the model submits a new implementation, it is scored across all possible inputs, leading to a perfect grading of the task. Since submissions are made, then improved, it produces a continuous score over time. These tasks have some undefined mathematical bound on how optimal they can be. Because the optimal solution is not known, and optimization is harder the more optimal the solution is, it is incredibly difficult to saturate. They can't be contaminated because there isn't a single correct solution to train on.

There are some drawbacks to this approach:

The relative ranking of models scores can be messed with when other model's transcript are trained on. However, the frontier of capabilities is not possible to fake, because there does not exist yet a better implementation to train on, so to do better is to generalize and genuinely be better at the task.

Some potential solutions: Because there's an easy to follow spec with examples, it may be easy to generate many different tasks that fit it. Whenever a new model is suspected of benchmaxing, generate a new small set of tasks and see if it still performs well. That way, there is no way of getting a good score without generalizing.

Memorizing solutions creates a different score curve than normal iterative improvement. For a model that has just trained on a frontier solution, it will be a single output that scores highly with no successful iterative improvements. Real solutions produce a score curve that looks roughly logarithmic.

For these scores, all models are run on the same harness, so the only difference is the model.