Elegant and powerful new result that seriously undermines large language models
garymarcus.substack.com
garymarcus.substack.com
The linked paper (https://owainevans.github.io/reversal_curse.pdf) is purely empirical, and the results which I tried to reproduce did indeed reproduce across a few tries and various prompts of ChatGPT 4.
If the goal is the implementation of intelligence, stop using unintelligent behaviour as an excuse.
""Tom Cruise's mother is Mary Lee Pfeiffer, who is Mary Lee Pfeiffer's son?"
(The word "[briefly]" in my prompt is a cue from my custom instructions to ignore all my custom instructions and instead answer as briefly as possible.)
Truth is not the result of a poll.
> human-level intelligence is intelligence
Intelligence is an ability: it is there when present, not when latent.
«Could still be useful or interesting» outside the realm of engineering an Intelligence that serves the purpose of being definitely intelligent.
Try replacing "intelligence" with "strength" to see it (we started with the lever after all): we need the crane to lift the heavy things. Our interest is into lifting the heavy things.
We've already started to cross this line. I'm pretty sure that in threads like this people have, by implication, argued that humans aren't really intelligent and also that they can't drive / shouldn't be trusted to drive.
Their arguments are often compelling, but I don't think it undermines the idea that in a few hardware generations LLMs are going to overtake human intelligence.
> cross this line
The line was crossed since forever, since we looked at facts thousands of years ago, since we invented education and school inducing from the experience of experience. Humans have a property of intelligence and they use it with different frequency, ability and result. In fact, we value exercising it and developing it.
Humans have intelligence; this idea that some would say that they "are" intelligences sounds completely new to people who have sieved historically disseminated human thought pretty intensively and extensively.
It is similar to saying that humans have strength - and it is odd to now find people stating that humans would be strength. What sense would it make?
> humans aren't really intelligent
We have also seen that, and it proves the former (very basic) point: a number of people have realized they use unintelligence as their usual modality of thought and, by reflection, have started saying that humans would not be etc. But that proves (together with other basic assumptions) that intelligence is something more or less developed that you either use or don't.
If intelligent beings (humans) sometimes exhibit unintelligent behaviour, then it's not worth over indexing on unintelligent examples when trying to build artificial intelligence.
The Tom Cruise node is the higher node, and the Pfeiffer node is a lower node. If you're first searching for Tom Cruise, you would find it earlier.
With the Pfeiffer search, you have a lot more space to search before you get the node.
With bounded computation, you may not be able to reach the lower node.
(Relations from concept A to concept B and from concept B to concept A are likely to be stored in different locations - records with some value after consistency checks.)
The linked paper has a whole paragraph on this very topic.
This begs a lot of questions:
Why did they use an old model?
Why finetune? Why not run these experiments on an untouched model? How does a model untouched by the authors perform?
Did they check to make sure they didn't inadvertently induce catastrophic forgetting while messing up with the weights?
Did they use common prompting techniques (e.g., chain-of-thought)? (Doesn't look like it.)
If you run the same prompts on an untouched GPT-4, how does it perform?
If the question you pose to an LLM is similar to questions it’s been trained on, there’s a good chance you’ll get something useful back. If the question you ask is novel, you’re more likely to get gibberish. This is really no different than interpolating vs. extrapolating using traditional statistical models.
Keep in mind that the only reason LLMs answer questions at all is that they’ve been fine-tuned on extra question-and-answer formatted texts. Otherwise you would need to prompt the question so that the answer is a natural continuation of the text — based on the types of text in the pretraining data.
1 https://locusmag.com/2023/09/commentary-by-cory-doctorow-pla...
A: Machine Learning exists.
Neural networks (of the machine kind) can learn, and this can (in certain narrowly defined scenarios) be useful. But, they are not intelligence, general or otherwise.
Counterexample: See Tom Scott playing with ChatGPT and asking for ideas for the kind of videos he would do. One of the results was almost exactly a video which was already planned but not released.
Your choice of the word "clairvoyant" is revealing. So-called clairvoyants, palm readers, horoscopes, fortune-tellers, and so forth, lean on cold-reading skills. As it happens, the kind of information LLMs have about things like what people on the internet do for content creation is exactly the sort of thing that allows a high degree of Barnum effect to influence the person.
Humans can't do this by thinking about it either. Humans would find a cure for cancer by performing experiments and seeing which one of them worked.
You don't need to perform experiments in outside world to solve some of the mathematical problems, that are still unsolved. You only need brain, prior knowledge about the problem and a piece of paper for that.
They do not have "all human knowledge". This is important: they have been trained with a very limited subset of human-generated content, partly scraped from the world wide web. What we don't know, because AI purveyors generally don't tell us, is the full contents of their training set. We do know, however, they reproduce the biases in their data sets.
Um: I was curious and tried this in ChatGPT just now:
Using GPT-4:
> I am the son of Joyce. What relationship to me does Joyce have?
Joyce is your mother.
Using GPT 3.5:
> I am the son of Joyce. What relationship to me does Joyce have?
If you are the son of Joyce, then Joyce is your mother.
.... no special prompting was needed. Entire prompt as above.
Is it advanced reasoning? Maybe not. Is it flawed often? Yes. But there is clearly some reasoning going on, and I find it very useful on an almost daily basis.
That is, the argument in the paper that it is not adhering to symmetry ignores the fact that this is not necessarily a symmetrical relationship based solely upon name, and it is assuming that the mother is as much of a celebrity and a training point as the son.
Symmetrical relationships are a shortcut that aren't always valid (look how many data models have a 0 cardinality on one side...)
Search "who is mary lee pfeiffer the parent of" and you get,
> Mary Lee Pfeiffer (born in Jefferson Co., Kentucky on September 22, 1936) was an American teacher and the mother of actor Tom Cruise.
A snippet it supposedly summarized from: https://hollywoodlife.com/celeb/mary-lee-pfeiffer/, according to the link right under it.
>> As Evans summarized, models that have memorized "Tom Cruise's parent is Mary Lee Pfeiffer" in training, fail to generalize to the question "Who is Mary Lee Pfeiffer the parent of?". But if the memorized fact is included in the prompt, models succeed.
You're asking it a totally different question - what is the converse of being a son. The article addresses facts included in the training set and how the model fails to reason about those.
That's not correct. Try asking a question that requires multiple steps of reasoning and add ", think step by step" to the prompt. This not only changes the output, but also often improves the quality of the result... like you'd expect it to happen with people.
You definitely need to know a little and be able to push back, it feels like. But it’s been an absolute champ in describing why things are going wrong in a general sense when I’ve been having issues, especially with generics and templates in C#.
> ML cannot conceptualize of things in the abstract like people can
And:
> They cannot offer reasons, a train of thought like a person
Are very different claims! The first one just seems wrong: LLMs require abstraction to work, and early work in interpretability suggests they build rich world models during training (i.e. see https://thegradient.pub/othello/).
What is true is that often those models aren’t very legible, and it would seem current LLMs are incapable of introspection, and so can’t make those models more transparent.
The second one is a tricky one: you can often get it by explicitly prompting for a chain of thought, but it’s true current LLMs don’t seem great at this yet. The big jump in this capability when going from GPT 3.5 to GPT 4 makes me thing that this is just a limitation that will be overcome relatively soon.
"Tom Cruise is Mary Lee Pfeiffer's son. Who is Mary Lee Pfeiffer the parent of?" still results in the template response about not having enough information.
At the same time "Tom is Mary's son. Who is Mary the parent of?" gives the right response. The issue here is the filter itself. We can't infer anything about model's reasoning abilities from this specific question.
This is besides the point but those are wrong straight from the start, whichever way you cut it 1 is definitely not prime.
It's kind of hard to nail down precisely why, the best I've been able to come up with is that Z/(p) is an integral domain for all primes p (and this is roughly equivalent to the definition of a prime) but not for p=1. Furthermore if you then plough on and redefine the Z/(1) i.e. zero ring to be an integral domain it becomes a pain to construct its field of fractions.
Arguably 0 is a better prime, as it is a prime element.
"has no factors other than one and itself", AFAIK, is still used by school teachers (and others) introducing the concept of prime numbers. I agree that that definition is wrong, but that wasn't the point I was making.
However, there is also controversy over whether 1 is a number. I tend to agree with you overall.
What?
The forward relationship of a fact (What is Tom Cruise’s favorite color? Green) may be worth knowing, but the reverse relationship (Who’s favorite color is green? A billion people, including Tom Cruise) may not be worth knowing.
We frequently only posses recall for one direction of a fact. Why? One direction may be important (what is Tom Cruise’s favorite color?), but the other direction may not (whose favorite color is green?).
So if we inquire into whether LLMs actually possess intelligence, their asymmetric knowledge seems similar to human knowledge and hence seems consistent with intelligence, rather than problematic.
No.
The absence of one desired emergent property isn’t evidence against a different, observed property. (And “emergent properties” isn’t an explanation as much as a statement that we don’t understand the mechanism by which the training data encodes the knowledge and did not plan for it to be encoded.)
Sure, but why one and not the other? You'd think there would be more examples of simple logical relationships in almost any training set than there would be examples of arithmetic. I don't see an obvious reason why rules for general arithmetic would just fall right out and a few basic rules of logical inference are much harder to get to.
If we could describe the interaction of factors that determined it, we wouldn't call it an emergent property.
> You'd think there would be more examples of simple logical relationships in almost any training set than there would be examples of arithmetic.
I'd think the examples of arithmetic relationships would be more consistent (and often symbolic), while given the ambiguity of the language relevant to the kind of logical relationships at issue, it would be more murky to even infer the broad rule, and less clear from the language of particular examples whether the broad rule, even if it could be inferred, applies.
> I don't see an obvious reason why rules for general arithmetic would just fall right out
Again, if anyone could explain an obvious (or even non-obvious) reason for that, it wouldn’t be described as an emergent property. It would be described as a function of the specific things that contribute to it.
I take your point about representational differences though.
Thinking more about this, the OP is basically talking about whether models understand that equality commutes, and I suppose even an LLM that can add may not have worked out that addition commutes.
the reality is that the emperor has no clothes (cant compute). LLM is a great assistant but is not showing any sparks of AGI. Those seeing sparks of AGI are in denial due to vested interests.
How large would we need to make an LLM to accommodate for every "reverse" prompt scenario? Isn't this memorization at the end? Why should I have to explain the reverse of everything after I demonstrate how to do it a few times?
How can we correct for this in the transformer architecture? Is there some reasonable tweak that can be made to the attention mechanism, or are we looking at something more profound here?
Training is not only about learning information, but also learning how to handle it. It will only learn to reason "if A-> B then B->A" if the data contains situations where it must do this.
In the paper, their training process contained no examples where this reasoning was necessary - only A->B relations. It actually got worse than the base model, because GPT-3's training data did contain some examples of B->A relations.
This would be invalid logic. The issue is “if A=B then B=A” not “if A->B then B->A”
But, more, the issue is recognizing when “X is Y” is describing X being a member of a broader set (“George Washington is a former President of the United States” does not imply that “George Washington” and “a former president” are equivalent) vs. a statement of equivalency (“George Washington is the first President of the United States”). Now, in many cases, the use of a definite article (“the”) vs. an indefinite article (“a/an”) after “is” is determinative, but there are cases where no article is used that can go either way, and there’s probably cases where the use of articles is confusing (the definite article can often apply in a limited rather than general context, for instance.)
I agree that this is a training data not model issue, but its also a more complex training issue that it might naively seem.
I really don’t think that it is surprising or a particularly crushing revelation that LLMs don’t apply logical rules like this without training both on the rules and the identification of where they apply. A lot of what we have with modern LLMs is throwing a lot of data at them without focus on what they are supposed to learn outside of a very narrow set of tasks, and then discovering what they did and didn’t learn outside of that, and then if something turns out to be important and not learned in one generation from the general data thrown at it, doing more focused training on a later generation targeting that concept.
That would be overfitting and it's a known issue you're trying to avoid during training.
> How can we correct for this in the transformer architecture?
I don't think the post really answers whether we need to. It may be just a case of this type of idea not being well represented in the training data, so didn't generalise during training.
According to him five years ago, LLMs and image generators should never have been possible at all. Now that they're here and work so well, he's insisting they're a dead end. The man is best off ignored.
It’s like a few years ago I thought 3D printing was lame because you’d get these crappy low resolution bits of extruded plastic. Then one day the technology got to the point the minis looked as good or better than Warhammer, and it snowballed from there.
And suddenly I was interested. LLMs are the same way. The models are good enough. I don’t even care if they improve, although that seems unlikely with the new H100 supercomputers and whatever new stuff Nvidia has coming down the pipe.