LLMs trained on “A is B” fail to learn “B is A”

If A then B. A. Therefore, B. -> Valid.

If A then B. B. Therefore, A. -> Not valid.

Even before computers we created formal languages (mathematics, logic equations) precisely because human language is too often ambiguous.

It's entirely possible there is nothing wrong with the logical reasoning abilities of LLM architectures and this result is simply an indication the training data doesn't provide enough infomation for LLMs to learn the symmetrical/commutative nature of these "is" relationships.

Though, based on the find-the-next-token architecture of LLMs, it seems logical that LLM should need to learn facts in both directions. If it's input set contains <Fictitious name>, it makes sense the tokens for "<fictitious album>" and "composer" will show up with high probability. But there is no reason that having the tokens "composer" and "<fictitious album>" in the input set should increase the probability of the "<fictitious name>" token, because that ordering never occurred in the training data.

If true, it would would suggest that LLMs have a massive bias against the very concept of symmetrical logic and commutative operations.

Jumping to conclusions like "if A then B" to "A=B" is a very common mistake for humans, bad statistics and propaganda. So I am actually positively surprised that models don't make that mistake.

The final puzzle piece is then recognizing the difference between the question "Who composed <x>" and "Who did <x> compose", one asking for the object of the passive sentence and one for the object of the active sentence.

In a "traditional" system without ML you would represent this with a directional knowledge graph <Artist> --composed--> <Album>, with the system then able to form sentences or answer questions in either arrow direction. But that conversion is generally tricky unless you know how many other arrows exist. That's obvious with categories, but even if you know that one person composed a song that doesn't tell you that only that person composed that song. That can lead to unsatisfying answers, and might be a reason why this is hard for LLMs.

• GPT4 (and other LLMs) is some kind of generalized homotopy engine. You can give it any input, ask it to apply any "translation". Language translation, style translation, or even keeping the style but talking about another subject, or translating code to another programming language – and it gives you something different, yet identical. "Write something like ... but ..." There is some deep understanding of what identity is here, in particular with respect to the messy expectations of our human sign systems: you can throw any kind of equivalence path, and GPT4 will handle them just fine. It seems the limit is not in its ability to generalize to any kind of identity schema we throw at it, but in the complexity of these schemas.

• I'm not saying GPT has an explicit understanding of these schemas/homotopies. My point is that even though GPT doesn't know much about homotopy type theory, I think it knows them in a latent way: GPT would perform much better at translating a piece of code in one language to another than it'd be at explaining what it just did in sound terms what through the lens of homotopy type theory. That knowledge about identity/equivalence is implicit.

The rest of my thoughts: https://pastebin.com/zSKHKqw3

Note: I'm not claiming to have a clear view of what's at stake here, just that there is a link between textuality, identity, and the foundations of logical inference

Having been involved in both traditional machine learning and common sense AI in my grad school years, I've seen first-hand the limitations of a purely statistical approach. (Some of my past data augmentation work is being used to benchmark LLM reasoning.)

While most folks are too fixated by the 'quick wins' achieved by LLMs the trade-off is often a lack of non shallow reasoning. And I worry that many active researchers are glossing over these deeply rooted issues.

The expectation has been set to human-level understanding and explainability by everyday common people who don't need to know about how it works. Given it lacks explaining basic logical deduction and even regurgitating its own mistakes, we can't even begin to compare this to humans as it is not the same.

We're just searching for the prompt that can reliably break these LLMs to show their lack of reasoning and at some point eventually someone is going to find it and it will break all of them.

Well, yes, narratives that look like reasoning and have accurate cobclusions are more common in their training data with langauge that references reasoning before them, so prompts that call for reasoning explicitly produce narration that looks like reasoning. (And which has more accurate conclusions, too.)

But not capable of doing logical deduction of the data it is trained on, just the data you give it.

It is very stupid about all the data in its training corpus, since it is encoded like a grammar and not a knowledge base.

This wouldn't be a problem if not for there being a million times more data in the trained set than what you can give it. But this mean that we can't really train the AI to be smart about a wide range of things, since the training corpus is stupid and the live data is very limited. (And it isn't even that smart about the live data, given how expensive it is to compute for that little amount of knowledge)

I checked just now a simple letter counting task, it's hilarious: https://chat.openai.com/share/4d4926b1-a6fe-4815-af77-4456c7...

The problem - to me - seems to be that the specific description is equivalent to one and only one person, and that is what the models seem incapable of learning; that for all x, y matching that particular description, x=y, and exists only one person matching said description.

Of course it’s a different question as to how exactly do humans understand chess.

I personally don’t believe humans have some sort of “special” intelligence, whatever we do must be computable and therefore doable by a computer. It’s just that todays LLMs don’t seem to be it.

Meaning that for computer programs they're very bad at chess.

I've been thinking about this and I think there is a simple trick to make LLM's play chess well. Any chess position can be represented by a string, like so: rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1 (Forsyth–Edwards Notation) If we think of a chess game as a sentence or a sequence of such strings or tokens, then after feeding tons of those to the model it is reasonable to suspect it will do what it does best, that is predict the next token. There is no spacial awareness or anything beyond that going on I think.

While LLMs are “merely” token predictors, it seems that perhaps the “knowing” is actually imbedded in the large corpus of memetic data.

In the same way that an infinitely detailed “choose your own adventure” book shows that data and computation are interchangeable by degree, (calculation vs lookup), my conjecture is that the often surprising effectiveness of LLMs, as well as there very human like flaws, are the result of this kind of computation-in-the-data scenario.

I posit that the overt connections in textual human knowledge also carry the implied knowledge that provides the unwritten context necessary to understand, if enough vector relationships can be teased out of a sufficiently large corpus of data.

It could easily be that our own “understanding” is also derived from the vast n-dimensional matrix that represents all human cultural knowledge.

Unfortunately, I’ve met several humans to which this statement would also apply.

So like a marketing department? Or most humans, most of the time, for that matter.

LLMs just predict tokens. It's truly amazing how much value you can get out of that, but we don't need to make more out of it than is necessary.

These models were trained on an unfathomable amount of text generated by real humans, we shouldn't be surprised that they sound convincing. But we also shouldn't confuse their ability to sound convincing with legitimate intelligence.

If anything, it seems like we're learning that the Turing test isn't a good predictor of actual intelligence. Mostly because real humans are really easily tricked into seeing intelligence where there is none (and paradoxically not seeing intelligence where there is).

Hard disagree. A subset of behaviours resembling or even "indistinguishable" from understanding should not be enough to meet the definition.

Maybe some day we will have AI systems that get there, but LLMs ain't it.

But you wouldn't call that learning as a human. If you include a chess manual in the learning corpus of an LLM, but not any examples of playing chess, then the LLM can give you all the rules about chess, but it can't play chess. That isn't learning, that is just parroting.

A human who can recite the rules of chess but can't play chess, would you say that he understands the rules of chess? No, he just know how to output the words and not perform the actions, so there is no understanding there. Same with the LLM, nobody can say it understands these things since you would never said a human understood under similar conditions.

For example, if you train an LLM with a chess manual but no chess games then the LLM can recite all the rules of chess, but it wont be able to play chess. That is what people mean when they say "LLM's are just token predictors, they don't understand anything".

So it has nothing to do with spirituality or consciousness or metal vs brains, its just a logical argument based on the limitations of just training the model to predict tokens to match data it has seen. Since it only tried to predict tokens it has seen, and it has never seen a chess game, it doesn't matter if it has seen all the rules, it was never trained to try to follow rules it sees so it can't play chess.