How easy is it for you to create an SVG of a pelican riding a bicycle in a text editor by hand?
Nobody's preventing them from rendering it and refining. That's certainly what we'd expect an AGI to do.
I didn't mean to imply it was simple, just that it's funny because I can't really evaluate evals like Humanity's Last Exam, but I can see the progress of these models in a pelican.
Without looking at the rendered output :)
And without ever seeing a pelican on a bicycle :)
It should Google it like the rest of us.
I'm surprised they haven't all tried to game this test by now, or at least added it to their internal testing knowing they will be judged by it.