New secret math benchmark stumps AI models and PhDs alike

Is this just what we would expect?

An LLM doesn't have access to the type of information needed to solve these problems so it simply can't?

"Many experts cite this as evidence that current large language models (LLMs) are poor generalist learners. "

That makes sense right?

It's not "learning" it's doing math on words to string them together, but without the right context / training, it doesn't know what words make sense to solve the problem?