Comment by lostmsu | Hacker News Reader

Comment by lostmsu | Hacker News Reader

Parent Full thread

> prediction which is closer to memorization

> don't memorize inputs - they predict them

I feel some tension here.

> rice grains on a chess board? Sure, but this has nothing to do with chess, and nothing to do with how many games were in the LLM's training data.

> just a list of 64 numbers

> remember even a few positions? Sure they could, but that's irrelevant.

I don't think you do. Or rather you do know the legend but for some funny reason seem to be unable to apply its lesson here, because you are talking about enormous terabytes of training data.

> Intelligent humans created the training data, and the LLM attempts to predict (copy) the training data, so of course it looks intelligent.

If for you it is about intelligence, I am out of this discussion.

You are talking about 2^64 being a huge number I assume ?

If not, then what are you talking about ?

If yes, then what is the relevance to an LLM playing chess ?

> just a list of 64 numbers

> remember even a few positions? Sure they could

A rough estimate of number of positions across all X move games is X^10. For 15 moves it is hopeless to remember even a relatively small part of them. Typical game has 40 turns, 1 move per player, so 80 moves.

1) The number of unique chess games that could theoretically be played (but mostly never have been), is irrelevant to what an LLM is remembering. It can only remember what was in it's training data - a far smaller number of maybe 10's of millions of games (of 30-50 moves each).

2) An LLM is not going to memorize vs generalize when there is no training pressure to do so. You might expect it to memorize book openings that occur over and over in the training data, but not some random non-celebrity game that occurs once in the Lichess dataset and is never again referred to.

> They can't possibly remember even a few positions. Don't you know the legend about rice grains on a chess board?

If the wise man was a bit wiser, he'd have asked for his rice on a snakes & ladders board (100 squares, not 64) and would have had 2^36 more rice, which is equally irrelevant.

Reply on news.ycombinator.com