Comment by martin-t | Hacker News Reader

Comment by martin-t | Hacker News Reader

Parent Full thread

I don't understand how anybody can still claim LLMs show "complex reasoning".

It's been shown time and time again that they'll produce a correct chain of reasoning when given a problem (e.g. wolf, goat, cabbage crossing a river; 3 guards and a door; etc.) that is roughly similar to what's in the training data but will fail when given a sufficiently novel modification _while still producing output that is confidently incorrect_.

My own recent experience was asking ChatGPT 3.5 to encode an x86 instruction into binary. It produced the correct result and a page of reasoning which was mostly correct, except 2 errors which if made by a human would be described as canceling each other out.

But GPT didn't make 2 errors, that's anthropomorphizing it. A human would start from the input and use other information plus logical steps to produce the output. An LLM produces a stream of text that is statistically similar to what a human would produce. In this particular case, it's statistics just weren't able to cover the middle of the text well enough but happened to cover the end. There was no "complex reasoning" linking the statements of the text to each other through logical inferences, there was simply text that is statistically likely to be arranged in that way.

I agree that LLMs don't really reason, but I'm starting to think that the name "Large Language Model" is just wrong enough to create this confusion. What they seem to be doing is modeling human reasoning as encoded in language, and then enable some mixing and matching of elements of that encoded reasoning.

This somewhat means that these models are trapped within the universe of "human capable reasoning" with some possibility of escaping it through the stochastic generative processes they're built on. But they simply can't think through novel problems and arrive at new conclusions.

Furthermore, they're limited by the fact they're built on human knowledge as encoded in text, which terribly imprecise and fluid. Any hope to have a path to AGI, where reasoning might actually happen, will have to have something far more rigorous for the internal reasoning, with language just being a clever interface rather than the mechanism by which the thinking is done in.

They're really "Large Analogy Engines" or maybe "Large Captured Reasoning Engines" and are an incredible technology.

GPT-3.5 is not the same thing. When people talk about LLMs having capabilities like complex reasoning, they're talking about current-gen models (eg Claude 3.5 Sonnet, GPT-4, Llama-405B).

>there was simply text that is statistically likely to be arranged in that way

This is wrong. An LLM can produce text that has never been arranged that way in training data.

Your statement doesn't contradict their statement. It produces text that is statistically likely to be arranged that way, but because we use non-deterministic sampling we get a wide variety of results that weren't necessarily in the training data.

No, I don't think they are talking about the decoding process there. If they were, it's a non statement. They are almost certainly talking about training data.

Perhaps it’s because I know human beings that have the exact same operation and failure mode as the LLM here and I’m probably not the only one. Failing at something you’ve never seen and faking through it is a very human endeavor.

Regarding errors: I don't know the exact mechanism in the brain that causes humans to make them but i believe it's a combination of imperfect memory, attention span and general lack of determinism. None of these affect logical reasoning as performed by a machine.

Regarding faking it till making it: This is a more general point that there's a difference between simulating human behavior and logical reasoning.

If you teach your kids to read, they behave almost exactly like an LLM in so many cases it's eery.

Reply on news.ycombinator.com