Hacker News Reader

That's not meaningless at all, it's a great metric! If the "cost per correctly-solved benchmark task" is infinity, you know not to use the model.

View on HN

They’re saying if the average task you actually use the model for is far less difficult than the benchmarks, you might incorrectly conclude that the model is costly when in fact it’s the best performing model for your actual use case.

I want a model that generates commit messages fast. Currently I have to wait up to a minute or two. That model doesn’t need to score very highly on SWE benchmarks, just highly enough that it can write out a good enough message in a few seconds. If you tested it on ${current top tier benchmark} you’d think it’s way too costly when in fact it’s the best tradeoff.

Your comment makes sense but I'm pretty sure yreg is saying the opposite of that—that their task is harder than the benchmark currently is.

(see their follow-up reply: "The cheapest-per-benchmark-task model would be useless to me if it cannot do the task I need.")

In either case, you need the right benchmark for the right task

As always, the relevance of any given benchmark depends on how similar what it’s testing is to your workload.

Sisyphus doesn't care about energy per meter to move the rock.

Reply on news.ycombinator.com