Hacker News Reader

Comment by jameshart | Hacker News Reader

Interesting to hear him say this:

> And I would never have thought that deep thinking could come out of a network that only goes in one direction, out of firing neurons in only one direction. And that doesn't make sense to me, but that just shows that I'm naive.

I think people maybe miss that LLM output does involve a ‘loop’ back - maybe even a ‘strange’ loop back, and I’m surprised to see Hofstadter himself fail to pick up on it.

When you run an LLM on a context and sample from its output, you take that sampled output it generated, update the context, and iterate. So the LLM is not just feeding one way - it’s taking its output, adding it to its input, and then going round again.

So I don’t think this implies what Hofstadter is saying about intelligence maybe being less complex than he thought.

View on HN

I was surprised he said this too. Even without the autoregressive part of GPT models you have a deep transformer with attention, so even a single forward pass can modify its own intermediate outputs.

The interesting thing though is that attention effectively allows a model to meta-learn based on the current context, so in many ways, it may be thought of as analogous to a brain without long term memory.

What do you mean? Attention is just matrix multiplication and softmax, it's all feed forward.

I wasn't disputing that it's feed-forward. I just meant that stacked transformer layers can be thought of as an iterative refinement of the intermediate activations. Not the same as an autoregressive process that receives previous outputs as inputs, but far more expressive than a single transformer layer.

Just being a part of any auto-regressive system does not contradict his statement.

Go look at the GPT training code, here is the exact line: https://github.com/karpathy/nanoGPT/blob/master/train.py#L12...

The model is only trained to predict the next token. The training regime is purely next-token prediction. There is no loopiness whatsoever here, strange or ordinary.

Just because you take that feedforward neural network and wrap it in a loop to feed it its own output does not change the architecture of the neural net itself. The neural network was trained in one direction and runs in one direction. Hofstadter is surprised that such an architecture yields something that looks like intelligence.

He specifically used the correct term "feedforward" to constrast with recurrent neural networks, which GPT is not: https://en.wikipedia.org/wiki/Feedforward_neural_network

It only yields something that looks like intelligence when you update the context and iterate though.

GPT can give a single Yes/No answer that indicates a fair amount of intelligence for the right question. No iteration there. Just a single pass through the network. Hofstadter is surprised by this.

Well, no, it can produce a probability distribution over all possible tokens, among which ‘yes’ or ‘oui’ or ‘はい’ or ‘totally’ or ‘no’ or ‘nein’ or the beginning of ‘as a large language model I am unable to answer that question’ are all represented. Which is either more or less impressive than just being able to answer ‘yes or no’ depending on your priors I guess.

There’s maybe an interesting philosophical question of perspective there because if you think of the GPT as answering the question ‘if you had just read this, what token would you expect to read next?’ That doesn’t seem like a question that necessarily requires ‘intelligence’ so much as ‘data’. It’s just a classification problem and we’ve been throwing NNs at that for years.

But if you ask the question ‘if you had just written this, what token would you expect to output next?’ It feels like the answer would require intelligence.

But maybe they’re basically identical questions?

The point of my comment is that even the distribution represents intelligence. If you give it a tricky Yes/No question that results in a distribution that's 99.97% "Yes" and negligible values for every other token, that is interesting. Hofstadter is surprised you can do any amount of non-trivial reasoning in a single forward pass.

Reply on news.ycombinator.com