Hacker News Reader

Comment by LoganDark | Hacker News Reader

LLMs don't think, and LLMs don't have strategies. Maybe it could be argued that LLMs have "derived meaning", but all LLMs do is predict the next token. Even RL just tweaks the next-token prediction process, but the math that drives an LLM makes it impossible for there to be anything that could reasonably be called thought.

View on HN

True. People use completely unjustified anthropomorphised terminology for marketing reasons and it bothers me a lot. I think it actually holds back understanding how it works. "Hallucinate" is the worst - it's an error and undesired result, not a person having a psychotic episode

A chess program from 1968 has "strategy", so why deny that to an LLM.

LLMs are built on neural networks which are encoding a kind of strategy function through their training.

The strategy in an LLM isn't necessarily that it "thinks" about the specific problem described in your prompt and develops a strategy tailored to that problem, but rather its statistical strategy for cobbing together the tokens of the answer.

From that, it can seem as if it's making a strategy to a problem also. Certainly, the rhetoric that LLMs put out can at times seem very convincing of that. You can't be sure whether that's not just something cribbed out of the terabytes of text, in which discussions of something very similar to your problem have occurred.

This is not a bad way of looking at it, if I may add a bit, the llm is a solid state system. The only thing that survives from one iteration to the next is the singular highest ranking token, the entire state and "thought process" of the network cannot be represented by a single token, which means that every strategy is encoded in it during training, as a lossy representation of the training data. By definition that is a database, not a thinking system, as the strategy is stored, not actively generated during usage.

The anthropomorphization of llms bother me, we don't need to pretend they are alive and thinking, at best that is marketing, at worst, by training the models to output human sounding conversations we are actively taking away the true potential these models could achieve by being ok with them being "simply a tool".

But pretending that they are intelligent is what brings in the investors, so that is what we are doing. This paper is just furthering that agenda.

> The only thing that survives from one iteration to the next is the singular highest ranking token, the entire state and "thought process" of the network cannot be represented by a single token, which means that every strategy is encoded in it during training, as a lossy representation of the training data.

This is not true. The key-values of previous tokens encode computation that can be accessed by attention, as mentioned by colah3 here: https://news.ycombinator.com/item?id=43499819

You may find https://transformer-circuits.pub/2021/framework/index.html useful.

This is a optimization to prevent redundant calculations. If it was not performed the result would be the same, just served slightly slower.

The whitepaper you linked is a great one, I was all over it a few years back when we built our first models. It should be recommended reading for anyone interested in CS.

People anthropomorphize LLMs because that's the most succinct language for describing what they seem to be doing. To avoid anthropomorphizing, you will have to use more formal language which would obfuscate the concepts.

Anthropo language has been woven into AI from the early beginnings.

AI programs were said to have goals, and to plan and hypothesize.

They were given names like "Conniver".

The word "expert system" anthropomorphizes! It's literally saying that some piece of logic programming loaded with a base of rules and facts about medical diagnosis is a medical expert.

rivers don't think and water doesn't have strategies, yet you can build intricate logic-gated tools using the power of water. Those types of systems are inherently interpretable because you can just look at how they work. They're not black boxes.

LLMs are black boxes, and if anything, interpretability systems show us what the heck is going on inside them. Especially useful when half the world is using these already, and we have no idea how t hey work

Water doesn't think, yet if you inject it into the entrance of a maze, it will soon come gushing out of the exit.

> rivers don't think and water doesn't have strategies, yet you can build intricate logic-gated tools using the power of water.

That doesn't mean the water itself has strategies, just that you can use water in an implementation of strategy... it's fairly well known at this point that LLMs can be used as part of strategies (see e.g. "agents"), they just don't intrinsically have any.

Reply on news.ycombinator.com