That's the basic definition of "the singularity".
Initially there might be the easy option of using more tokens/memory/<whatever easily measurable and beneficial factor you can imagine>. But when that becomes impractical, how will the "dumber AI" select between x generated "smarter AIs"? How will it ensure that the newly generated versions are better at all (if there are no easily measurable parameters to be increased)?
Something that scares me is people don’t want to believe this is where we’re headed or it’s even possible. I say this because I think your concerns are easy enough to address, it makes me think you didn’t try answering your own questions.
The idea in this scenario is it’s self optimizing. No reason it can’t make it’s own more specific tests to the general test of “become as smart as possible”. And people can make tests that they’re not smart enough to pass. You just make them as a team and take more time. It could also discover new things and then verify if they’re true, which is easier.
I think I answered that. You use more resources to create it than answer it. When creating it, you can have a team take lots of time, use all sorts of tools, etc. When taking it, you constrain time and tools. That’s the general pattern of how to make a question harder than you can answer.
There are also lots of trapdoor questions, especially in math. An overly simple example is factoring numbers.
There also might be thresholds, barriers, were small increments in intelligence are not possible. But that is speculative, I will admit.
I’m not saying I’ve solved the path to AGI or anything. It’s just that this specific issue of “making a test harder than you can solve” has an answer.