Hacker News Reader

Comment by LPisGood | Hacker News Reader

The thing is that they are completely incapable of meta-cognition. Reasoning models don’t show their actual reasoning at all.

View on HN

Right — they're not reasoning, they're generating text that statistically models reasoning. Anyone who says differently is selling something.

That is what a base model does. After RL it is a very different thing, and anyone who says they know what it is, is naive or dishonest. These things are grown, not made, and we really do not understand how they work in many important ways.

Yeah, but they’re not magic; we can still do experiments and see what happens. Anthropic did a lot of work on this and showed that they’re not accurately describing their reasoning process.

Of course, the fact that they have to do that proves my point.

Humans are also not magic; we also do experiments on humans, and humans also do not accurately describe our reasoning process.

Lot of confabulation going on in the moist blob of electrochemistry found encased within the hydroxyapatite crystal cage we call a skull. Why is it that things which rhyme, or get repeated a lot, seem more true? How come some of us suffer uncontrollable seizures from flashing lights?

We have to study reason to get any good at it, we absolutely suck at this without training. Our natural state is illiterate, innumerate, and illogical.

The part of our (all animals, not just humans) intelligence that is most magical*, is how efficient we are with few examples, not reasoning.

* Questions about consciousness will have to wait until we can agree which of the 40+ definitions of the word actually answers the question we care about, and then also we figure out how to actually test for whatever that is.

As the meme goes, "they are the same picture".

Language has reasoning encoded within it.

It certainly does. But so too do complex neural network functions, as do attention mechanisms.

Reply on news.ycombinator.com