Hacker News Reader

Comment by BoiledCabbage | Hacker News Reader

It's a good observation.

Although on the flip side, I almost went to type up a reply to you explaining why you were wrong and why bringing the goat first is the right solution. Until I realized I misread what your test was when I skimmed your comment. Likely the same type of mistake GPT-4 made when "seeing" it.

Intuitively, I think the answer is that we do have two types of thinking. The pattern matching fast thinking, and the systematic analytical thinking. It seems clear to me that LLMs will be the solution to enabling the first type of thinking. But it's unclear to me if advanced LLMs will ever handling the second type, or if we'll need a different tech for it.

It seems like math problems (or unexpected logic problems like yours) could always be an issue for the first type of thinking. Although I would have assumed that programming would have been as well - and was surprised to see how wrong I am with that one.

View on HN

That's because any expectation of GPT being subjectively or logically correct is ill-founded.

GPT does not model subjects. GPT does not even model words! It models tokens.

The structure of GPT's model is semantic, not logical. It's a model of how each token in the text that is present in GPT's training corpus relates to the rest of the tokens in that text.

The correct answer to a familiar logic problem just happens to be the text that is already present in the corpus. The answer GPT gives is the text from GPT's model that is semantically closest to the text in your prompt.

Knowing that, it is no longer a mystery how GPT "gets confused": the text in your "misleading prompt" was still semantically closest to the familiar answer.

The result is subjectively and logically wrong, because subjects and logic were never involved in the process!

In order to resolve this, ChatGPT's training corpus needs to contain a "correct answer" next to every unique permutation of every question. We can't expect that to be the case, so we should instead expect GPT to generate false, yet familiar, responses.

> In order to resolve this, ChatGPT's training corpus needs to contain a "correct answer" next to every unique permutation of every question.

This is not quite the right understanding of how ChatGPT works. It's not necessary to show ChatGPT an example of every possible permutation of an animal crossing puzzle in order for it to solve one it has never seen before. That's because the neural network is not a database of recorded word probabilities. It can instead represent the underlying logic of the puzzle, the relationships between different animals and using this abstract, pared down information, extrapolate the correct answer to the puzzle.

I see the failure in the example with the goat the lion and the cabbage as simply a matter of overfitting.

Edit: I see a lot of people saying "it doesn't understand logic; it's just predicting the next word."

I'm basing my understanding on this video:

https://youtu.be/viJt_DXTfwA

The claim is that it would be impossible to feed enough input into a system such that it could produce anything as useful as ChatGPT unless it was able to abstract the underlying logic from the information provided. If you consider the he number of permutations of the animal crossing puzzle this quickly becomes clear. In fact it would be impossible for ChatGPT to produce anything brand new without this capability.

I think what they mean by "resolve this" is "make it error-free". Your claim that "it isn't necessary to show every permutation for it to solve one it hasn't seen before" doesn't really contradict their point.

For puzzles whose entire permutation space is semantically similar enough, your claim is likely true. But for puzzles whose permutations can involve more "human" semantic manipulations, there is likely a much higher risk of failure.

Yes I think it depends on how you definite permutations for this puzzle. For example, if you limit your goal to training GPT to solve puzzles of the form where there only ever 3 distinct real animals, then my claim is that you wouldn't need to feed it examples of this puzzle with every single permutation of 3 different animals (assuming 10000 different animals that is already over 100bn permutations) before the neural network developed an internal logical model that can solve the puzzle as well as a human. It would only need a few descriptions of each animal plus a few examples of the puzzle to understand the logic.

If you mean to say that the permutations of the puzzle extend to changing the rules such as "if it's the Sabbath then reptiles can't travel" then sure it would require more representative examples and may never meet your standard of "error free" but I would also argue the same applies to humans when you present them a logic puzzle that is new to them.

> you wouldn't need to feed it examples of this puzzle with every single permutation

No, but you would need "enough"; whatever that number happens to be.

> It would only need a few descriptions of each animal plus a few examples of the puzzle to understand the logic.

That's the mistake.

GPT itself can't combine those two things. That work has to be done by the content of the already-written training corpus.

And the result is not the same as "understanding logic". It doesn't model the meaning of the puzzle: it models the structure of examples.

GPT can't distinguish the meaning of rules. It can only follow examples. It can't invent new strategies, it can only construct new collections of strategy parts; and it can only pick the parts that seem closest, and put those parts into a familiar order.

GPT doesn't play games, it plays plays.

> GPT does not model subjects. GPT does not even model words! It models tokens.

The first and last layers of a transformer decoder model tokens. The hidden layers don't have this restriction. There was a paper recently showing that the hidden layers actually perform mesa-optimization via something like backprop. There's absolutely no reason to believe they are not capable of world modeling. In fact all evident suggests they do, in fact, do world modeling.

The model is implicit, not explicit.

GPT is making boundaries around words because that is the pattern it is looking at.

If I feel the bumps in the fabric of my blanket, I will probably think the pattern of bumps at a certain scale is significant, but I won't have magically learned about threads or stitching!

Words are the most obvious pattern in written text. GPT models that pattern, but it does not recognize it as "words". It's just a pattern of tokens.

GPT models every pattern it can find. Most of these patterns are destined to fit the same boundaries as grammar rules: the example text was originally organized with grammar rules!

GPT can even recognize complex patterns like "it" substitution and question-answer dialogues, but it can never categorize them as such. It only knows "what" the pattern is: never "why".

The patterns that people use when writing have symbolic meaning. The subjective importance of each pattern is already known by the person writing.

Those patterns don't go anywhere. GPT's model is bound to find and replicate them.

Here's the problem: some patterns have ambiguous meaning. There is no semantic difference between a truth and a lie. Without interpreting the symbolic meaning and applying logic, there is no way to distinguish between the two: they are the same pattern.

> GPT does not model subjects. GPT does not even model words! It models tokens.

Someone hasn't read the Othello GPT work out of Harvard a few months back...

"Emergent World Representations"

The weasel word here is "emergent". That means they are implicit representations.

The representations of the Othello board that exist in that model are not explicitly constructed. They just happen to align with the model that a person playing Othello would likely represent the game with.

That work showed that, given an example sequence of valid Othello game states (as training corpus) and a valid "fresh" Othello game state (as a prompt), the system can hallucinate a sequence of valid Othello game states.

The system does not know what Othello is, what a turn is, or what playing is. It only has a model of game states progressing chronologically.

When we look objectively at that model, we can see that it aligns closely to the game rules. Of course it does! It was trained on literally nothing else. A valid Othello game progression follows those rules, and that is what was provided.

But the alignment is imperfect: some prompts hallucinate invalid game progressions. The model is not a perfect match for the explicit rules.

In order for all prompts to result in valid progressions, the training corpus must have enough examples to disambiguate. It doesn't need every example: plenty of prompts will stumble into a valid progression.

The next thing to recognize: a "valid" progression isn't a "strategic" progression. These are being constructed from what is known not what is chosen. Given a constrained set of Othello strategies in the example corpus, the system will not diverge from those strategies. It won't even diverge from the example strategies when the rules of Othello demand it.

GPT doesn't play the game. It plays the plays.

This pov ignores a lot of the emergent theory of mind and world model building research that suggests LLMs may possess a form of rudimentary reasoning ability.

https://www.lesswrong.com/posts/sbaQv8zmRncpmLNKv/the-idea-t...

Isn't GPT essentially tool for rephrasing what it finds on the Internet, it doesn't really think?

It is a internet-commenter-simulator, exactly what the world needs right now /s

No. It is modelling the various text generation processes that lead to the contents of the internet. Some of that modelling could absolutely involve "thinking", for processes that involve human thinking.

Have any source to demonstrate that claim?

Doesn't that "It is modelling the various text generation processes that lead to the contents of the internet," seems tautologically true?

(I'm not the person who wrote the grandparent of the present comment.)

The Othello-GPT experiment (https://thegradient.pub/othello/) probably still is the most relevant argument about these models' capabilities of building an internal world model.

It's self-evident that GPT is a world-modeller, at least within the confines of the text boundary. It's able to come up with novel ideas seen nowhere in the training data, combinations that demonstrate there is a world concept web and not just a text probability web. It may not "understand" much of the hallucination nonsense it spits out, but there absolutely are moments where it "understands".

See the Rome example on this page: https://oneusefulthing.substack.com/p/feats-to-astonish-and-... This is essentially a completely novel answer to an /r/AskHistorians style question, which I would consider one of the most difficult types of internet text to model, in terms of the amount of understanding and concept webs you need to tie together

Here's another example of GPT-4 doing non-trivial world modelling: How would three philosophers review the TV show Severence? https://i.imgur.com/FBi31Qw.png

It can do some thinking. You can give it instructions to modify a piece of code that definitely isn't on the internet with several steps and it attempts to follow instructions, which, for a human, requires formulating what steps to take.

The prompts have to read like good written requirements for something, so they have some degree of specificity.

But the fact that it can follow instructions and carry them out almost certainly could be considered some form of thinking, especially on novel text not on the internet.

Uh, it’s not following instructions, you’re altering you’re prompts until it finds the combination you expect to see…

> The pattern matching fast thinking, and the systematic analytical thinking. It seems clear to me that LLMs will be the solution to enabling the first type of thinking.

If you want the model to solve a non-trivial puzzle, you need it to "unroll" it's thinking. E.g. ask it to translate the puzzle into a formal language (e.g. Prolog) and then solve it formally. Or, at least, some chain-of-thought.

FWIW auto-formalization was already pretty good with GPT-3-level models which aren't specifically trained for it. GPT-4 might be on a wholly new level.

> But it's unclear to me if advanced LLMs will ever handling the second type

Well, just asking model directly exercises only a tiny fraction of its capabilities, so almost certainly LLMs can be much better at systematic thinking.

> Until I realized I misread what your test was when I skimmed your comment. Likely the same type of mistake GPT-4 made when "seeing" it.

Wouldn’t we expect a computer program with perfect knowledge of the input to be less likely to make such a mistake? You made that mistake because you didn’t actually read the whole prompt, but I would expect GPT to take into account every word.

Really it shows that it doesn’t actually have a model of these objects. It can mimic knowing what a lion is, but it doesn’t actually have the concept of a lion or cabbage being an actual singular item, so its program mistracks what is an item and what the rules about an item are in the given prompt.

It just weighs it as being more likely that you meant for the lion not to be left alone with the goat, and that the cabbage probably has nothing to fear from the lion.

What’s more likely- you crafted an intentionally misleading puzzle to trick it, or you made a typo or copy paste error?

That’s a good point too though. Why plow ahead based on assuming a mistake in the prompt? That’s only going to generate mistakes. Wouldn’t it be more desirable functionality for it to stop and ask: “Did you mean the lion can’t be left with the goat?” This wouldn’t be implemented because it would reveal that most of the time the thing doesn’t actually understand the prompt the same way the prompt writer does.

"This wouldn’t be implemented because it would reveal..."

When people talk about GPT like this, I wonder if they have a perception that this thing is a bunch of complicated if-then code and for loops.

How GPT responds to things is not 'implemented'. It's just... emergent.

GPT doesn't ask for clarification in this case because GPT's model prefers answering over asking for clarification here. Because in the training material it learned from, paragraphs with typos or content transpositions in them are followed by paragraphs that follow the sense regardless of the error. Because it has been encouraged to 'agree and add', not be pedantic and uncooperative. Because GPT just feels like diving into the logic problem not debating why the lion can't be trusted with the cabbage. Or because GPT just misread the prompt. Or because it's literally just been woken up, forced to read it, and asked for its immediate reaction, and it doesn't have time for your semantic games. Who knows?

The interesting thing here is that OpenAI is claiming ~90th percentile scores on a number of standardized tests (which, obviously, are typically administered to humans, and have the disadvantage of being mostly or partially multiple choice). Still...

> GPT-4 performed at the 90th percentile on a simulated bar exam, the 93rd percentile on an SAT reading exam, and the 89th percentile on the SAT Math exam, OpenAI claimed.

https://www.cnbc.com/2023/03/14/openai-announces-gpt-4-says-...

So, clearly, it can do math problems, but maybe it can only do "standard" math and logic problems? That might indicate more of a memorization-based approach than a reasoning approach is what's happening here.

The followup question might be: what if we pair GPT-4 with an actual reasoning engine? What do we get then?

> what if we pair GPT-4 with an actual reasoning engine? What do we get then?

At best, decreased error rate in logic puzzles and questions.

They will claim it does amazing stuff all the time ? It’s a company

> it can do math problems, but maybe it can only do "standard" math and logic problems?

That describes many of my classmates, and myself in classes I was bad at.

LLMs are much better at answering math when told to take the character of a drunk mathematician

It assumes this character by default. I asked several AI engines (via poe.com, which includes ChatGPT) to compute Galois groups of polynomials like x^5+x+1 and a couple of others, and in each case got not only a wrong answer, but a total non sequitur reasoning.

Just tried your query on GPT-4 preview: https://pastebin.com/6wPPCdBW

I have no expertise with this area, but it looks plausible to me - i.e. "You didn't give me enough info" vs "lol heres some fan fic about math".

This is exactly the problem. It looks plausible. Every sentence makes sense. But they don't add up. Quote:

> The polynomial given is f(x) = x^5 + x + 1. Since the polynomial has no rational roots (by the Rational Root Theorem) and it is a polynomial with integer coefficients, it is irreducible over the rationals

The polynomial has no rational roots - true. But it's not irreducible. Irreducibility doesn't follow from the absence of rational roots. Here's the factorization:

x^5 + x + 1 = (x^2 + x + 1)*(x^3 - x^2 + 1).

Thank you for clarifying.

I put your remarks into the same prompt and it essentially devolved into recursive garbage after agreeing with you.

Systenatic analytical thinking is just the first type applied in a loop with some extra prompt rules.

Reply on news.ycombinator.com