The exclusion of harness's feels really weird given that companies are recognizing the value of what harness's can do. By excluding them the benchmark is becoming less relevant.
Similarly LLMs are not just massive uniform artificial neural nets, and now we also have harnesses, which I'd personally view more of an extension of the model itself. The harness is both the executive and also what allows it to keep notes, use a calculator, or maybe even create scripts to help it solve complex deterministic problems.
I think it's unfair to give a human a very complex maths problem and say that they're not intelligent if they can't solve it without pen and paper or a calculate. At least expecting a human to solve complex problems this way doesn't really measure anything useful in the real world.
I think in this age where models are heavily RL-ed on acting in specific harnesses, this type of benchmark is more important than ever, to make sure they’re not in fact moving further away from general intelligence.