It'll be interesting to explore how robust it is for statements crafted so that they surely did not occur in the training data, but their truth value could be indirectly inferred from information in the training data.
More generally, the organic training data is almost always incomplete. Samples of text leave many assumptions and inferences hidden. Like a math problem, you see the statement, but don't know the answer until you work it out. Or like a puzzle, you see the pieces but don't know the big picture until you fit those pieces together.
That is how training text samples are like unsolved enigmas, we train our models on undigested text. Often the pieces are spread over many training examples that almost aways appear separately, never together have the chance to draw a conclusion from them. Search is needed, augmenting training examples with supporting data.
Neural nets are smart at inference time but dumb at training time. They don't make those connections when they train. Instead, we need to draw those connections out by generating new text. We need to benefit from inference-time smarts before training. That means we need to use current LLMs to write the dataset of next LLMs.
All the best LLMs today used a big piece of synthetic data, including GPT-4. Datasets like Orca, Phi-1.5, ShareGPT, etc. It's also the best way to create small models (<10B) that actually work, you need very high quality, high diversity synthetic data.
It does not mean $cows === $mammals.
"A is a B" Or "A's are B's" means that A is included in B.
$a === $b ("A is B") implies that $b === $a (B is A), in all cases that I know of.
https://owainevans.github.io/reversal_curse.pdf
"In particular, suppose that a model’s training set contains sentences like “Olaf Scholz was the ninth Chancellor of Germany”, where the name “Olaf Scholz” precedes the description “the ninth Chancellor of Germany”. Then the model may learn to answer correctly to “Who was Olaf Scholz? [A: The ninth Chancellor of Germany]”. But it will fail to answer “Who was the ninth Chancellor of Germany?” and any other prompts where the description precedes the name."
(Which isn't necessarily a evidence against truth models for out-of-distribution facts, it could just be a matter of indexing.)
So the issue is that they cannot infer that general rule due to a fundamental limitation of the transformer LLM architecture, not just a training data issue? I skimmed the paper and it seems to be the case.
I get the overall idea, but this statement isn't always true, right?
"The sky is blue" does not imply "blue is the sky".
Is that really so? It would seem to be well within what I thought they were capable of based on all the other things they can do correctly.
That said, I think people make too much of it as an "LLMs can't reason" point, when I don't think that's accurate. What it says is that LLMs instant recall is not logically bidirectional, but this is something that humans do as well. Humans take longer to respond to (and are less accurate answering) "Who is Tom Cruise's mother?" than "Who is [her name]'s son?". At least for me, when I get questions that are the "wrong way around", I have to literally run through it logically in my head, generally along the lines of "(What does that name remind me of, is she a spy? Or her son is a spy? Is she a fictional character? Wait I think this is is a celebrity thing, which spy celebrity has her as his mum? Oh yeah, Tom Cruise.) [Out loud:] Tom Cruise."
Also, some people misunderstand the actual deficiency, and think that the LLM can't answer the question at all, rather than just zero shot. The LLM can answer the question if it has the information in context, it can reason "If A=B, then B=A" just fine. It just can't do the less popular halves of AB equivalencies zero shot.
The paper is, apparently, still under review.
In the mean time, may I suggest you to verify that example by yourself?
Me: Who is tom cruises mother?
ChatGPT: Tom Cruise's mother is Mary Lee Pfeiffer.
User: Who is Mary Lee pfeiffers son
ChatGPT: Mary Lee Pfeiffer's son is the famous actor, Tom Cruise.
Interestingly though, you can give it additional clues and it'll get it. https://chat.openai.com/share/893c1088-6718-4113-a3f1-cf273d...
I also agree with him about humans capable of the same "errors".
Adrian Tompkins was the ninth mayor of the town of Wolverhillington. Who was the ninth mayor of Wolverhillington?
The ninth mayor of the town of Wolverhillington was Adrian Tompkins.
"Adrian Tompkins was the ninth mayor of the town of Wolverhillington"
And later ask in inference,
"Who was the ninth mayor of Wolverhillington?",
It might not return the answer.