This reminds me of "McDonald's Theory":
https://jonbell.medium.com/mcdonalds-theory-9216e1c9da7d
24 karma · joined November 13, 2013
This would bring another dimension to it since then quality of tokens would be one dimension (RTS-language: Decision Making) and speed of tokens the other (RTS-language: Actions Per Minute; APM).
Also there are a lot of coding benchmarks, that way it would test something more abstract, similar to AlphaStar https://en.wikipedia.org/wiki/AlphaStar_(software)
You could just use the exposed APIs of OpenAI, Anthropic etc. and let them battle.
The "sometimes" you talk about should just be with ads on. it won't hurt much, because it's just sometimes.