One-shot benchmarks are great for me as a solo creator, since they slightly correlate to whether the better frontier models (Opus and Fable for me) make better decisions about things I didn't spec, or whether they'll give me better suggestions right off the bat.

just found a decent looking benchmark for iterative development: https://swe-milestone.com/

surprised it isn't a bigger thing, eg artificial analysis doesn't report anything like that

still doesn't measure the human-agent interaction part, but that's pure vibes atp

It's not, but it's trying to bring any level of objective measure in this realm, vs just going off of vibes.

Yeah, the models have all been really good at generating greenfield apps for a really long time (in the scope of LLM time).

I suppose it’s interesting to see how they make better greenfield apps. But I am much more interested in how they solve hard problems in existing gnarly codebases.

I imagine one could one-shot a basic app and then feed feature requests one by one, sounds like an obvious way to benchmark architecture/maintainability

Absolutely yes, but that's how you become twitter/X famous

Man I do ponder this all the time.