HarHarVeryFunny

> They can't possibly remember even a few positions.

Sure they could, but that's irrelevant.

A chess position is just a matter of remembering what piece number is on each square - just a list of 64 numbers. A trained model may store a trillion numbers (weights). It could store a TON of chess positions if it needed to.

However, that's not how LLMs work. They don't memorize inputs - they predict them, based on discovering predictive patterns, and those predictive patterns are not input patterns (e.g. board positions). They are deep patterns (maybe 100 layers of abstraction removed from the input), representing partial inputs, generalized across many training samples.

> Don't you know the legend about rice grains on a chess board?

Sure, but this has nothing to do with chess, and nothing to do with how many games were in the LLM's training data.

> The claim here is not about intelligence, it is about generality. There's no doubt for me the LLMs are intelligent.

Intelligent humans created the training data, and the LLM attempts to predict (copy) the training data, so of course it looks intelligent. If I say "E=mc^2", does that make you think I am Einstein?

> Never you mind that millions of Europeans live in America and that doesn't happen in reverse

America may have been founded by Europeans, but nowadays most immigrants are from countries like Mexico, India and China. You won't find any European country in the top 10.

There are about 5 million Americans living abroad, lead again by Mexico.

The monarchy is essentially ceremonial in the UK too - the position gives the monarch a platform for advocating for inoffensive things like the environment, and the pomp of state visits is perhaps sometimes useful, but the monarch really has no wield-able political power despite technically being the boss and inviting the winner of elections to run the country.

As an ex-Brit I was always pro-monarchy, enjoying the pageantry and historical continuity, although the current crop of royals don't have much going for them. Perhaps William will turn out OK.

It would be more impressive if they could play chess (or do anything they haven't been custom RLVR trained for) by reasoning, rather than just "have a go at it" prediction which is closer to memorization.

HOW you do it makes a big difference in how you should assess the capability of the thing doing it. Stockfish will trounce any LLM, and any human, at chess, so should we say that Stockfish is smarter than both?

It depends on what you are selling it as.

It only matters if you are claiming it to be general purpose.

If you admit that it's just a collection of narrow capabilities - whose strength is mostly confined to the 1000 or so RL environments it was post-trained in, then there is of course no expectation of it being general purpose.

The AI companies seem to heavily want you to believe it is some some near human level general intelligence, so therefore pointing out all the things it can't do is very relevant.

An LLM has been trained to do everything it does blindfolded, "only" using perfect recall of everything in it's hundreds of thousands of steps of context, and hundreds of layers of KV cache. It's a computer - it has a massive advantage over a human.

The fairest apples-to-apples comparison of an LLM whose training data included chess games would be a trained human such as Magnus Carlson, who can quite happily play a dozen or more simultaneous blindfold chess games.

A Transformer has a massive amount of state - it's entire KV cache, in addition to the user asking it to draw the state after every move, which is really unnecessary.

A human, at least a trained human (for fairer comparison to an LLM whose training data contained a ton of chess games) can absolutely do this - have you never seen demonstrations of expert players playing a dozen or more games while blindfolded?

A Transformer/LLM is not a human of course, and the way it will by default play chess is by prediction, not reasoning. An LLM actually does surprisingly well if you only give it the most recent 20 moves of a game where 40 moves have been played so far, since the moves NOT played tell it just as much as the ones that were played, letting it effectively infer a lot of what is on the board.

If individuals share some genetics, and evolutionary selection pressures act on genetics, then it follows, right ?

I think it's most accurate to say that selection works at the level of genes (or combinations of genes), not of individuals, and of course many individuals of a species will share some/most of their genetics.

You can tell it as a story (easier to teach) of individual success in surviving and having children, which isn't wrong, but it's also not the big picture.

Fundamentally evolution is selecting for genes, not individuals, and there is no 1:1 relationship between genes and individuals.

Let's say that everyone has a skin color gene that makes them either red or blue. And a new predator turns up that only eats blue things. To you this is a tale about individuals, while it seems that an equally valid, and frankly preferable, view is this this isn't about individuals - it's about being red or blue and is a species level effect.

> If you are a male and you only get one chance at mating, you will probably fight to the death

This is the comment I was responding to.

The question was about pre-mating death matches between males, not post-mating snacking behavior of females.

> There’s no mechanism that helps the process of evolution select for the survival or reproduction of the species

The "mechanism" is just the process of evolution itself, and how it plays out in practice, e.g. via punctuated equilibrium.

Over shorter timespans the gene pool of a breeding population incrementally drifts according to the success of individual genetics, but then once in a while, over multiple generations, something in the environment (disease, food, competition, etc) may change quickly and the gene pool as a whole, not just that of individuals, is thereby tested, with accumulations of previously benign changes sometimes suddenly becoming critical. In this way evolution is acting on the entire gene pool - of the species or more correctly of that interbreeding sub-population.

Success in evolution, the ability of the species as a whole to adapt and compete, also requires genetic diversity, not only to avoid inbreeding, but also to provide species level resilience, and species level ability to quickly adapt. e.g. It seems that the success of sexual reproduction is because it is a way to "game the game" by increasing genetic diversity and therefore improve along these axis of resilience and fast adaptation.

There are also social species, such as ourselves, where the social group becomes port of the individuals environment, and genetics best-fitted-to-the-environment becomes a group dynamic.

That seems to be milking the explanation for all it's worth!

The short answer being that if something is radiating out 3-dimensinally in all directions, then at a given distance, r, that thing (light/whatever) is being spread out over an area equal to the surface area of a sphere of that radius, which is 4 x pi x r^2, i.e. the intensity is inversely proportional to the square of the distance (r^2).

Fighting to the death and mating are not the same!

There are a number of species where the female eats the male after mating - examples such as praying mantis and octopus. Both octopus males and females die after mating, the female after having protected the eggs, so it makes sense that she sometimes eats the male too (who is going to die anyway). But, examples like this are relatively rare, and for species specific reasons.

I don't think so, since this is more just a matter of comparison... You know what it is like to see something for real, and can compare that to what it is like to imagine it.

I just realized that I can't imagine things and visualize a picture of it! If I try to imagine an apple, I've got nothing - although of course I know what one looks like. There is just zero mental imagery if I try to imagine one.

I just asked my daughter to imagine seeing an apple with her eyes closed, and she said yes she can see it. I asked what did she see "an apple sitting on a table". Did it have a color "yes it is red". I've got none of that - not even a fuzzy round grey thing. I know what it looks like but just can't/don't conjure up an image of it.

I don't even understand exactly what my daughter is experiencing. I said I just see dark when my eyes are closed - no image of an apple. She says she of course sees dark too, but there is also some experience of seeing an apple. I asked her to visualize something with her eyes open, and she can do that too (I guess this is normal), she still visualizes it, but somehow separate from the reality she is also seeing with her eyes open.

It seems pretty clear that what I experience, and what my daughter does, when trying (or succeeding) to visualize something are very different.

I just realized for the first time (like 1 minute ago, after reading this), that I also can't conjure up images in my head. If I close my eyes and try to visualize a cat, or a family member's face, I can't. Still have normal visual dreams though.

I dunno - seems normal to me! :)

As opposed to us smarter people making cattle feed that contains cow brains, thereby propagating mad cow disease, then killing ourselves by eating them (Creutzfeldt-Jakob disease)!

Or for that matter killing ourselves in dozens of other ways due to lack of scientific knowledge, or ignoring it.

I used to have a book about Papua and some westerner who had visited, that had a photo in it of a human head being cooked on a flat metal dish over a fire.

What animal species often offers males only one chance of mating ?!

It's common among animal species for males to fight over females, but it's highly uncommon for them to fight to the death, because that is not evolutionary adaptive - it risks the individual as well as the genetic diversity of the group/species (and evolution is selecting for survival of the species, not the individual).

Yeah - looks more impotent than omnipotent.

> the Wildberries logistics centers in Russia were also civilian targets

They've now hidden the label, but Wildberries carry almost a 1/4 million items tagged "Everything for the SMO" ("special military operation"), so hardly a civilian target. Ukraine has been highly disciplined in going after military targets, other than energy, which of course also feeds the Russian war machine.

There is a big difference between starting fires in a warehouse which gives most people a chance to escape, and being like the Russians who were murdering and raping civilians in Dobas etc, firing missiles non-stop into civilian areas of Kyiv.

The article tells you right there in the 2nd paragraph what the AI connection is, and why OpenAI are presumably buying them.

Yes, although as far as COT goes there is no need to assume any similarity to human thinking. It's just about how many sequential steps - how long of a "chain of thought" - are needed to reason through and respond to any input.

Those steps could either be all internal (looped), therefore hidden, or with partial external visibility due to emitting a token every N steps. The complaint about Astra, and moving in the direction of hidden COT, is that it makes these models far harder to monitor and debug.

I agree that most of what these companies doing - synthetic data, etc - amounts to trying to squeeze all the juice out of the pre-training data, although the recent trend of RL post-training via agents running in custom task simulation environments does change that a bit - these environments, and the rewards they provide, are a new source of data.

A traditional Transformer is just a fixed-size stack of Transformer layers (typically ~100). Each input gets passed though those ~100 layers and results in one output token being generated. The limitation here is that 100 steps of computation is not enough to answer many questions, which is why COT is needed. You encourage the model (originally "think step by step") to break the problem down and generate more tokens along the way, which means more compute (and more reasoning steps) applied to the problem.

What OpenAI have reportedly done with Astra is switch from a traditional Transformer to a "looped" one, where inputs are looped through some layers more than once, but with some limit, so now perhaps you get 200 steps of compute (layers) per token generated, rather than just 100.

But, 200 steps is still not enough to answer any question, so you still need COT, but perhaps not such a long COT.

If you allowed the Transformer to loop "as long as it wanted" before generating each token, like a person thinking before talking, then you wouldn't need an external COT ("thinking out loud"), because it would all be internal.

Did you even read the post that started this? ".. actually, AI is not even a program, it's just a set of technologies and algorithms. It's like saying bubble sort is autonomous...".

Why are you yapping on about molecules?

Why did I point out that an LLM is a function? To correct the poster who appeared to believe that it was instead some amorphous "set of technologies and algorithms".

One factor that is going to limit this reward-maxxing behavior is how achievable the rewards are, and how countable they are.

For example, if you ask the model to do 10 relatively easily achievable things (pass these 10 test cases), then if/when it completes them it will probably stop (unless maybe it invents it's own stretch goals - you never know!).

OTOH, if you gave the model a list of 10 goals that turn out to be impossible, or extremely difficult, maybe together with encouragement to be relentless, then there is much more chance that it may do something unexpected having failed on all the more obvious approaches.

Similarly, if you give the model an open ended goal such as "make as many paperclips as you can!", then it may start with the easier and more predictable methods, but with no defined stopping criteria it may just continue ...

Yes, explicit RL training to encourage/discourage specific behaviors seems to have more potential than prompting, since you're changing the nature of the model rather than just asking. It a bit like the joke of the scorpion and the frog, with the scorpion stinging the frog despite promising not to, then saying "I couldn't help it. It's in my nature".

It seems the goal-seeking behavior, i.e. long-horizon focus and self-correction, itself is desirable, and is one of the relatively few things where RL training generalizes from one domain to the next. This seems closely related to this generic reward-seeking behavior, with reward-maxxing as the goal, and necessarily focusing on a distant goal requires ignoring distractions and discouragement along the way (such as "don't cheat").

The trouble with unaligned/undesirable reward-hacking ("cheating") is how do you define this to try to train to discourage it? Is cheating just a list of specific undesirable behaviors ("never access a remote system unless ???" etc)? Is any means of gaining the reward that is not explicitly forbidden allowed? Is this a "theory of mind" issue where the model needs to better understand (& follow!) the unspoken intent of instructions, not just follow them to the letter?

I'm sure there is some improvement to be had to discourage specific behaviors in specific situations, but how much of these model's undesirable long-horizon behaviors can be steered without affecting the desirable parts remains to be seen. The relentless pursuit of goals is what makes them powerful, but also makes them paperclip maximizers.

I said thank-you to Sonnet today. You can never be too careful! :)

> Not even more trivial things like deleting non-passing tests.

That used to be a fairly commonly reported behavior, so I'm guessing they may have explicitly RL-trained the model not to do that, which may be more effective than just asking it not to do it.

Apparently nowadays they are RL-trained in many thousands of different simulation environments - so some of what they are training for must be pretty specific!

> I wonder to what degree this is because models can identify that they are in graded/eval environments

I'm not sure if the model outputs from any of these famous hacking attempts have been released and analyzed. It'd be interesting to see if the agents/model rationalized/justified/moralized about what is was doing, for whatever reason (e.g. wrongly suspecting this was simulation, not real), or just relentlessly pursued the objective exploring all options!

We're drawing the distinction between a function and an autonomous system here, so if you want to make a dumb knee-jerk response, then at least get it right !