I think we are still far from nailing down good LLM benchmarks, because the more general-purpose your software the harder the question of what makes it good becomes. Is Python a good programming language? Is Java? Is C? I think it's a similar class of problem. You can benchmark rudimentary things like execution speed similar to how you can benchmark tokens/second, but these metrics don't tell the whole story.