Language isn't thought. It's a representation of thought.
Probably there are at least two groups of people and neither really comprehends how the other thinks haha.
Spatial thinkers, for example, or the hyperlexic.
Meaning for hyperlexics is more akin to finding meaning in the edges of the graph, rather than the vertices. The form of language contributing a completely separate graph of knowledge, alongside its content, creating a rich, multimodal form of understanding.
Spatial thinkers have difficulty with procedural thinking, which is how most people are taught. Rather than the series of steps to solve the problem, they see the shape of the transform. LLMs as an assistive device can be very useful for spatial thinkers in providing the translation layer between the modes of thought.
A thought does not capture every process that went into producing the thought either.
A painting of a landscape can capture details of the landscape it's representing. The landscape itself doesn't capture anything, it just is.
Sure, but the landscape is something, namely an aggregate of particles. A thought in principle isn't its physical expression, but its information content, and it's represented in a human brain by some aggregate of particles. So no matter how you slice it, thoughts can only manifest within representations, and so calling language a representation of thought isn't some kind of dunk, because it also proves that human brains don't have thoughts.
It's not clear whether the information content of all possible human thoughts can be captured by language, but clearly at least some language expressions have the same information content as human thoughts.
Most people have had no exposure to even the most basic ideas of language philosophy.
The idea all these people go to school for years and don't even have to take a 1 semester class on the main philosophical ideas of the 20th century is insane.
One must imagine Sisyphus happy and Chomsky incoherent with rage.
Otherwise they couldn't solve math problems that aren't simple rephrasings of problems they were trained on, and they obviously can do that. If you give a multi-step undergraduate level math problem to the human operator of a Chinese room, he won't get very far, while an LLM can.
So that leads to the question: given that they were trained on nothing but language, and given that they can reason to some extent, where did that ability come from if it didn't emerge from latent structure in the training material itself? Language plus processing is sufficient to produce genuine intelligence, or at least something indistinguishable from it. I don't know about you, but I didn't see that coming.
Seriously?