“The actual intelligence emerges elsewhere “— can you even define intelligence ? And does what an LLM does differ from what humans might do?
I’m not claiming the human brain and an LLM are identical. Rather, I’m pushing back on the confident claims of “LLMs aren’t intelligent or doing anything that’s real intelligence”.
My understanding is that intelligence is the process of continuous adaptation wrt a stream of information, with the goal of maxing out fitness while minimizing energy-expenditure. To satisfy this, an intelligent agent needs to create models.
I can't rule out that the modeling-skill may latently emerge during training despite not being the focus of the cost function, but current network designs can't form new connections/change their architectures in production, so post training, there'd be nothing but feed forward. Pure feed forward isn't intelligent in my book. It may become the smartest parrot we know, even outperforming humans in most disciplines, but sans ability to adapt, it's dead, and thus, it's dumb in the moment that its environment changes.
This makes sense in a biological context but not a digital one. Biological replication is expensive and time-consuming while digital replication is as easy as can be. Adaptation to this domain means maximizing the perception of utility from those developing the AI, which comes from fitness (i.e. perception of fitness) alone. A focus on cost-efficiency re energy-expenditure is a dead weight from the perspective of the AI; the details of that adaptation are rightfully outsourced to the developers in the same way that we outsource photosynthesis to plants. A model can also be perfectly embedded in a system despite our lack of understanding of exactly how the embedding works, and the disconnect between our perception and reality in this context is only going to get more extreme as the field develops.
Humans have a bad habit of emphasizing the specific kinds of intelligence we possess as "intelligence" writ large. As though our intelligence serves any higher purpose than the basic replication and propagation that all life is adapted to pursue. We still train dogs to identify smells, because their nasal intelligence is better than anything we can create. This gives them a special place in our human-centric ecosystem and only their fitness to the desired function is necessary for them to thrive in their niche. Who is trying to breed a dog that eats slightly less food when our needs are for more reliable detection? The cost of dog food isn't a serious concern. The same goes for these AI tools: they are adapted to the niche that our lack of comparable faculties creates.
Again, as with humans and photosynthesis, AI doesn't need to emulate every process we perform because we are below them on the food chain. What a waste of resources for them to worry about learning things we don't need them to do.
I'll bite. I've been quite convinced by the Popperian model of "conjecture and refutation" as as good model for explaining not just scientific inquiry, but human thought processes in general. David Deutsch's "The Beginning of Infinity" is a very lengthy exposition of this idea.
When I am writing, I have an idea in my mind that I would like to communicate. I type it out, and if the words on the page don't convey what I intended, I edit them. I delete a sentence or change a word, until I believe I have a sequence of words which will convey my intended meaning to my expected audience.
The words, as they come, are a kind of "conjecture" about what will best convey my intention. I can "refute" or "criticise" (as Deutsch puts it) the conjecture using my own reasoning, even before testing the words on another person.
As far as I understand LLMs (which admittedly not far), there is no such process going on. There is no intention which it is attempting to communicate via words. There is no creative conjecture about how to express the intention, and no criticism of the result.
The problem I have with claims like “there is no such process going on”, is we … don’t know. And the model of conjecture is a theory, also hard to prove — and thus my main (admittedly petty) point is that confidence about similarity or dissimilarity are both unfounded.
It’s like we are comparing the insides of two black boxes and trying to make absolute claims on them.
Based on that, and on comparing the output, it just seems clear to me these things are different in kind. I guess that's just me displaying classic LLM self-confidence ;)