Comment by stillpointlab | Hacker News Reader

Comment by stillpointlab | Hacker News Reader

I have to be careful of confirmation bias when I read stuff like this because I have the intuition that we are uncovering a single intelligence with each of the different LLMs. I even feel, when switching between the big three (OpenAI, Google, Anthropic) that there is a lot of similarity in how they speak and think - but I am aware of my bias so I try not to let it cloud my judgement.

On the topic of compression, I am reminded of an anecdote about Heidegger. Apparently he had a bias towards German and Greek, claiming that these languages were the only suitable forms for philosophy. His claim was based on the "puns" in language, or homonyms. He had some intuition that deep truths about reality were hidden in these over-loaded words, and that the particular puns in German and Greek were essential to understand the most fundamental philosophical ideas. This feels similar to the idea of shared embeddings being a critical aspect of LLM emergent intelligence.

This "superposition" of meaning in representation space again aligns with my intuitions. I'm glad there are people seriously studying this.

LLMs don't think, nor are they intelligent or exhibiting intelligence.

Language does have constraints, yet it evolves via its users to encompass new meanings.

Thus those constraints are artificial, unless you artificially enforce static language use. And of course, for an LLM to use those new concepts, it needs to be retokenized by being trained on new data.

For example, if we trained LLMs only on books, encyclopedias, newpapers, and personal letters from 1850, it would have zero capacity to speak comprehensibly or even seem cogent on much of the modern world.

And it would forever remain in that disconnected positon.

LLMs do not think, understand anything, nor learn. Should you wish to call tokenization, learning, then you'd better call a clock "learning" from the gears and cogs that enable its function.

LLMs do not think, learn, or exhibit intelligence. (I feel this is not said enough).

We will never, ever get AGI from an LLM. Ever.

I am sympathetic to the wonder of LLMs. To seeing them as such. But I see some art as wonderous too. Some machinery is beautiful in execution and to use.

But that doesn't change truths.

You made a lot of bold assertions there. It's as if you have a complete and definitive theory of human intelligence to compare it against. Which if true, would be incredible, because there isn't a scientifically accepted theory, nor is there consensus from a philosophical standpoint.

I can't say that you are wrong, you might be right, especially about AGI. And I think it's unlikely that LLMs are the direct path to AGI. But, just looking at how human brains work, it seems unlikely that we would be intelligent either if we used your same reductionist logic.

An individual neuron doesn't "think" or "understand" anything. It's a biological cell that simply fires an electrochemical signal when its input threshold is met. It has no understanding of language or context. By your logic, since the fundamental components are just simple signal processors, the brain cannot possibly learn or be intelligent. Yet, from the complex interaction of ~86 billion of these simple biochemical machines, the emergent properties of thought, understanding, and consciousness arise.

Dismissing an LLM's capabilities because its underlying operations are basically just math operating on tokenized data is like dismissing human consciousness because it's "just electrochemistry" in a network of cells. Both arguments mistake the low-level mechanism for the high-level emergent phenomenon.

When an LLM can re-tokenize on the fly, due to newly learned data, let me know.

It won't prove intelligence, but at least it won't be static like a book.

https://x.com/sukjun_hwang/status/1943703574908723674

> Tokenization has been the final barrier to truly end-to-end language models.

> We developed the H-Net: a hierarchical network that replaces tokenization with a dynamic chunking process directly inside the model, automatically discovering and operating over meaningful units of data

Good to see. Will be better to see once the scope is clear.

But something such as this is required to move towards actual intelligence.

This is a dumb critique. A thin wrapper running a new samples of training data and updating weights is something already done in many situations. A sophisticated rag system incorporates new information even if the weights themselves aren't updated, effectively giving "new memory".

LLMs have problems, in practice being "static" aint one of them.

I agree with much of what you say, but this:

> An individual neuron doesn't "think" or "understand" anything. It's a biological cell that simply fires an electrochemical signal when its input threshold is met.

is a wildly oversimplified representation of what a neuron is and how it interacts with the rest of the brain. Even what we know is far beyond this description and there's a vast amount more that we don't know.

One of the assumptions that people make (with or without realising they are making it) is that the brain is a componentised information processing device, that you can break it down into units that effectively communicate in a digital way. This may be true, but it may not. It may not be possible to decompose a mind into parts communicating via digital signals without sacrificing conscious experience.

Personally, I think it's likely that it isn't possible to do that, but I don't know for sure. Others clearly think it's likely that it is possible, which is a perfectly valid opinion. Neither should pretend that science has yet given us an unambiguous answer on the question.

LLMs don't think, nor are they intelligent or exhibiting intelligence. ....

---

My 2$: If you replace 'LLMs' with 'Humans,' most of your statements above still make sense — which suggests AGI might be possible.

Do you think an average human from 1850 transported to the modern world would fare better than an LLM?

This isn't a serious question, is it?

A human would immediately start learning, and understanding the differences. Forming new memories. People learn every, single day.

An LLM would never have a single new memory, nor understand a single change had happened. Without being retokenized via new data it trains on, nothing of its symbolic world view would even change in the tiniest bit.

Telling it so in a context window isn't the LLM learning, that's gone the second the context window is gone. In 5 years, 10 years, 1M years, that LLM would be precisely the same. It would view the world as 1850. Forever.

Meanwhile, being told "It's 2025!" would immediately update the human's expectations. They would see the changes around them, and start learning. Note that initial reaction isn't relevant here, a person would eventually deal with it, and move forward.

You have posted a link, in another response, where someone is working on dynamically updating an LLM. However that is not how current LLMs work, thus invalid in any attempt to refute how I describe LLMs currently.

Further, until we see how this works -- precisely, it may be a non-starter. It may not work as described. It may be hype. It may work, but still require retraining if large amounts of symbolic data needs to change.

There's a reason it takes massive farms of hardware to tokenize, and I'm skeptical that the quoted twitter link reforges all relationships. Again, we'll see.. and this is something required for an LLM to actually "learn" about changes to how tokens are weighted.

For long term memory, LLMs can learn new information by training on it. It would need a lot of training data, but it will get there eventually.

For short term memory, they can store information in their context window. Yes the context window is finite, but they can still be very large, and there exist techniques to fit more information within that window.

For medium term memory, there's not one single solution, but anecdotally it seems like solutions exist. My workplace has a chatbot that can ingest documents and save slack conversations, and then use the knowledge from those to answer questions indefinitely into the future. I haven't looked into how it works, but empirically it functions well across thousands of people for more than a year.

Between a human and an LLM transported 200 years into the future, a human will certainly have better critical thinking skills, but I'd bet an LLM system designed for the task will probably be able to more quickly learn to have coherent and symmetric conversations about the world with modern humans.

Overall, there's a lot that's unclear, and I think I'm pretty pessimistic about progress towards AGI, but I don't think we can deny that LLMs have something that is, at minimum, the best substitute for human intelligence that exists.

I disagree with you but I'll only repeat and expand on my final statement: I'm glad there are people seriously studying this. If you are right (which I doubt) I hope it comes to light with evidence. And if you are wrong, I hope you have the grace to admit it.

Reply on news.ycombinator.com