Hacker News Reader

Any metric that can be targeted can be gamed

Then target it with metrics worth solving[1].

But that seems to be measuring "superintelligence" rather than just AI, no?

useless benchmark if all it shows will be fail right; At least it's a very lagging benchmark

It's common for benchmarks to start at zero and eventually become saturated. The atari games benchmark started as such and is now a solved problem.

If the metric is a latent variable summarizing subjective judgements, yes.