I don't think this is true.
They have to express themselves as tokens. The meaning of those tokens doesn't have to be text. See any model that can handle images/video. Also, I don't think math, svg, etc, are "natural" language.
And, only the final expression is tokens. The intermediate layers, with the encoded concepts, aren't "natural language".
But, to address your concern (which nobody can disagree with, since even humans can't fully express through text/pictures), potentially: https://news.ycombinator.com/item?id=49758615
The model isn't limited to concepts that can be expressed in natural language.
It's only once the AI gets to the output layers that natural language comes back into play.
After all, they're all made out of weights[0].
How do we know for sure? We don't even know how the emergent properties we see actually emerged?
For humans we know for sure that people sometimes have concepts that they have no word for (the reason the phrase "It's on the tip of my tongue" is a phrase, after all).
We don't know this for LLMs. When it makes new phrases, it's always a mixup of two existing words hyphenated (aside, that also seems to be the limits of SOTA models creativity - join two unrelated words together with a hyphen).
LLMs never respond with "It's on the tip of my tongue" type responses, indicating it has a concept but cannot remember (or does not have) a word for that concept. Every human, pre-speech-age, has managed to express or convey concepts that they had no word for.
So, no. I'd need a citation, preferably multiple, that did the trials and found that a model can generate concepts for which it does not have any words for.
Even if the input is in plain English, the model never sees any words, tokens or glyphs to begin with. It's vectors all the way down.
1. Is natural language holding LLMs back by some %? 2. Is natural language serving as a hard gate that will prevent LLM intelligent progressing past some specific point?
The answer to 1 seems like an obvious yes to me.
Your thesis says the answer to 2 is "yes." That doesn't feel right to me. Think about all of the humans who have pushed various fields forward: Einstein, Newtown, Bach, whoever. If natural language doesn't prevent an entity from surpassing humans in one intellectual field, why would it prevent an entity from surpassing humans in all intellectual fields?
(To be clear, I'm not claiming superintelligence will or won't be achieved; I'm considering your specific thesis about whether or not natural language will be a hard gate)
By the way, how good is Claude's Hopi?