Imagine that I am building a translation app: one way to do it would be simply a dictionary database where the Czech word "pták" is related to the English word "bird"- we certainly agree that this app does not understand what a bird is at all.
But when I say I do understand what a bird is, what does that mean? The way I imagine it, my brain is wired in such a way that when the word "bird" occurs, a certain pattern of neurons that encode stuff like "living creature", "flying", "feathers", "eggs", and "beak" also fire (each of those is itself a pattern of other associations).
In this sense, I think LLMs can "understand": connections between words "bird" and "eggs" or "flying" have weights closer to 1 while connections to words like "iron" or "multiplication" have weights closer to 0. A mental model of a bird can be represented as a set of vectors in multi-dimensional space.
Maybe I am missing something but I see no fundamental reason why we would not be able to encode complex mental models like chess or liberalism or flat earth theory as sets of vectors (and maybe even determine the set of operations that can move someone from flat earth model to globe model in a similar manner as those examples with 'King – Man + Woman = Queen').
Again, I am not saying I know how the human brain works. All I am saying is that I think we can plausibly explain a lot of it in terms of just networks of units that have some weights attached to them - it is a very powerful concept. That is why I am asking when people seem to be assuming that there is more to "legitimate intelligence".
Also - and I say this with a high level of insecurity because I am no expert - I do not think that LLMs actually play chess the way you describe it. If I even understand correctly what you mean - are you saying that LLMs are only able to play moves they have "seen" in their training sets? If that is the case, I would disagree - I think there is evidence that LLMs have some ability to generalize, which means the ability to work with abstract patterns (which I would be tempted to call mental models).