Comment by ekidd | Hacker News Reader

Comment by ekidd | Hacker News Reader

Parent Full thread

> LLMs have no real understanding

A large language model trained on Othello games will actually build a "board representation" inside its neural network: https://thegradient.pub/othello/. This was confirmed by editing the board representation inside the neural net, "tricking" it into "believing" the board was in a different state. At this point, the LLM started making legal moves based on the edited position. If the LLM is actually building a data structure describing the board state, then it's hard to claim it's just acting as a parrot.

The raw version of GPT apparently achieves an Elo score of 1800 at chess: https://twitter.com/GrantSlatton/status/1703913578036904431. This is worse than a good dedicated chess engine, but almost certainly better than I could do!

And every day, I see Copilot occasionally perform non-trivial completions that require "in-context learning" of complicated things that exist only in my source code. Sure, it crashes and burns regularly, too, but I've seen a lot of cases where it could only have figured out the code it did by carefully combining information available in the file.

So I believe that LLMs can actually achieve surprisingly large amounts of understanding, particularly ChatGPT 4 and (on a good day) CoPilot. But the frustrating thing is that they're inconsistent, and the current RLHF process apparently impairs their ability to correctly estimate accuracy. When they make mistakes, they're effectively "blind" to those mistakes, to a shocking degree.

>> The raw version of GPT apparently achieves an Elo score of 1800 at chess: https://twitter.com/GrantSlatton/status/1703913578036904431. This is worse than a good dedicated chess engine, but almost certainly better than I could do!

Probably because it was trained to get better at it.

https://twitter.com/GrantSlatton/status/1706545268009931175

There's already been plenty of discussion on LLMs' ability to play chess and it makes sense that OpenAI would try to make their models better at it.

I’d point out that the LLMs are still incapable of using these internal representations in novel ways. A human who has internalized the rules of chess can learn and play variants like duck chess easily. GPT-4, despite being somewhat proficient at chess, falls apart when asked to play duck chess, constantly making illegal or illogical moves.

Uhh not really. I assure you, a LLM that can play chess will converge in training on duck chess much faster than one that wasn't.

LLMs are failing on tasks as simple as this. you can call it a trick question but what the hey.

"prove that there are no non negative numbers less than 3"

Claude puts out this elaborate BS - whats funny is the pretension.

----

This is trivial to prove by contradiction:

Assume there exists a non-negative number n < 3. Then either:

n = 0. But 0 is not less than 3, so this is a contradiction. 0 < n < 3. But the only numbers between 0 and 3 are 1 and 2, and neither of those are less than 3, so this is also a contradiction. Therefore, our assumption that such an n exists must be false. By contradiction, there are no non-negative numbers less than 3.

In formal logical terms:

Assume ∃n ∈ R, n ≥ 0 and n < 3 (There exists some non-negative number n less than 3) Then either: a) n = 0. But 0 ≥ 3 is false, so this is a contradiction. b) 0 < n < 3. The only numbers in (0,3) are 1 and 2, neither of which is less than 3, so this is also a contradiction. Therefore the assumption in (1) must be false. By contradiction, the statement ∀n ∈ R, n ≥ 0 → n ≥ 3 (For all non-negative numbers n, n is greater than or equal to 3) must be true. So the proof by contradiction shows that there are no non-negative numbers less than 3.

----

boosters can hype all you want, but the damn thing cant compute. and please dont say - "but most humans will fail at this too"

meanwhile...: claude-coders working furiously to add this test case - lol.

I swear, these kind of gotchas are the most stupid. Because they are easily resolved by just talking to the current state of the art.

https://chat.openai.com/share/93324013-b333-44ef-b8c8-5460cb...

ok, here you go. another slight tweak to a standard puzzle and it falls.

btw, i am very impressed with how well chatGPT does on this puzzle thanks to the elaborate tuning that seems to have been done by the training.

--- A farmer has a wolf, a goat, and some cabbage - and he must cross a river by boat with his items. The boat can carry only the farmer and a single item. If left unattended together, wolf would eat the cabbage, or the wolf would eat the goat or the goat would eat the cabbage. How can they cross the river without anything being eaten? ---

To ensure that nothing gets eaten during the river crossing, the farmer can follow these steps:

Step 1: The farmer takes the goat across the river and leaves it on the other side.

On the other side: Farmer | Goat On the original side: Wolf | Cabbage Step 2: The farmer returns alone to the original side.

On the other side: Goat On the original side: Farmer | Wolf | Cabbage Step 3: The farmer takes the cabbage across the river.

On the other side: Farmer | Cabbage On the original side: Wolf | Goat Step 4: The farmer leaves the cabbage on the other side and takes the wolf back across the river.

On the other side: Cabbage On the original side: Farmer | Wolf | Goat Step 5: The farmer takes the wolf across the river.

On the other side: Wolf | Cabbage On the original side: Farmer | Goat Step 6: The farmer returns alone to the original side.

On the other side: Wolf On the original side: Farmer | Goat | Cabbage Now, all three items (wolf, goat, cabbage) and the farmer have safely crossed the river without anything being eaten.

---

GPT4 understands that it's an odd variation of the classic puzzle, and that its unsolvable. (This conversation is my first attempt) https://chat.openai.com/share/5fe4f781-ea18-41ae-8ff9-349a31...

Will AI skeptics ever get tired of being BTFO by stochastic parrots?

Gary Marcus' Twitter timeline suggests not.

the input is a data structure describing board state.

i don't think uncovering the internal representation of board state, after providing the board state, implies any kind of phenomenal understanding - it just implies the software may accept input.

side-channel editing of the internal state, and observation of the resulting effect, does not imply understanding. you can do the same thing to any traditional game algorithm.

As I understand it, the Othello LLM was:

1. Trained using written sequences of moves from real world games, in text notation.

2. Prompted using written series of moves.

So it has supposedly never "seen" an Othello board state at all, if I understand the experiment. It deduced the rules of rules of Othello and "learned" a neural network containing a board representation.

It's entire possible that I misunderstood the paper, though? Why did you conclude that they were training and/or promoting using board state? (I should go re-read the paper; it's been a while.)

No you're right.

Clearly the information of the board state is implicitly present in sequences of moves (humans can play blindfold chess too but poorer) but a reconstruction from that alone is highly non trivial. I don't think a human that had never seen or played the game would be able to do it.

Agreed. But Othello syntax happens to map perfectly to possible Othello moves, so the most efficient representation of the information contained within Othello happens to be a data structure which maps to an Othello board. Other information like visuals on the players would, from the point of view of evaluating the next Othello move, just be noise.

Human language doesn't map perfectly or even particularly closely to the physical world we inhabit or emotions we experience though, so a maximally efficient model of human language will overfit to useless semantic features whilst lacking context on what foo that follows baz is. Understanding the physical or emotional world solely from human language is more like trying to use the Othello LLM's state representation to establish the colour of the board

The sense data your brain ingests does not map perfectly or even particularly closely to the real physical world we inhabit.

Whether it’s through language, vision (which feels quite “real” but is really just a 2D projection of light that we interpret), sound or anything else, it’s all just some byproduct of the world that we nonetheless can make useful predictions with.

There is enough information to build "a" world model in all the text ever written by humans about the world. Not necessarily the "one true model" that is your own personal life experience of the world since birth.

>Understanding the physical or emotional world solely from human language is more like trying to use the Othello LLM's state representation to establish the colour of the board

Nobody truly understands the physical world. Don't you think the birds that can feel the electromagnetic fields around the earth and use it to guide their travels would tell you your model was fundamentally incorrect ?

Certainly, LLMs are more limited in their input data, but it's not a fundamental difference. and adding more modalities is trivial.

> The sense data your brain ingests does not map perfectly or even particularly closely to the real physical world we inhabit.

I never argued otherwise, though being aware that there is a physical world that I can interact with helps! The point is that the only reason the LLM's transformation of syntax approximated an Othello board was the unusually perfect correspondence between permutations, syntax and efficient storage that seldom exists. In other circumstances your LLM vectors are modelling language constructs, lies and other abstractions that only incidentally touch on world and brain state.

The term "understanding" is generally used by humans to refer to how humans model reality[1] and need not imply completeness. But it also implies that a model isn't extremely dissimilar to humans in what its parsed and how its parsed it. Or to slightly alter your example, if a bat argued that following the bat swarm well enough to locate the exit didn't mean humans had achieved "true echolocation", I'd have to agree with them.

I mean, a photograph and a pocket calculator are also representations of some aspect of the state of the world, sometimes even representing a particular subset of the world information in sufficient fidelity to allow humans to make much better predictions about it. But fewer people seem to wish to stan for the capacity of the calculator or the bitmap to have "real understanding" of the outputs they emit, even though fundamentally the LLM has much in common with them and far less in common with the human...

[1]the potential for debate around such definitions underlines the paucity of language...

I think you underestimate humans.

As the other commenter said, this is incorrect. The input was a sequence of legal moves (not even "real" moves - most of the training data was synthetically generated with "generate legal moves" as the only constraint).

Deducing board state from this is extremely non-trivial.

Reply on news.ycombinator.com