> All representational schemas involve a compression of information about something, but what gets left in and left out in the compression varies. The representational schema of language struggles with more concrete information, such as describing irregular shapes, the motion of objects, the functioning of a complex mechanism or the nuanced brushwork of a painting — much less the finicky, context-specific movements needed for surfing a wave. But there are nonlinguistic representational schemes which can express this information in an accessible way: iconic knowledge, which involves things like images, recordings, graphs and maps; and the distributed knowledge found in trained neural networks — what we often call know-how and muscle memory. Each scheme expresses some information easily even while finding other information hard — or even impossible — to represent.
That said, we haven't reached the limits of language models yet. No one knows for certain how language models that are 10x or even 100x larger than current state-of-the-art ones will perform at modeling language (e.g., as quantified by perplexity) and at harder cognitive tasks.[a]
--
[a] https://www.gwern.net/Scaling-hypothesis#scaling-hypothesis