Multimodal Neurons in Pretrained Text-Only Transformers
huggingface.co
huggingface.co
That said, I keep thinking these posts are about electricity.
"Deep conversation"- I like how dang puts it: curious conversation.
The "neural net" started out as an analogy for a particular kind of statistical model. Now that analogy is being mistaken for some sort of fundamental truth.
And if you suggest otherwise, it's the technologists who have it right, not the neuroscientists.
Do you mean that in the abstract sense, or do you know what patterns are those, and when they "emerge"? Could you describe them? For example, can you give a mathematical formula for them?
I can't count the number of times I've seen people argue that LLMs are proof that the human mind is just a statistical model.
Meanwhile, any neuroscientists will be the first to tell you that we know surprisingly little about how neurons actually work together within the human brain, and certainly don't understand how consciousness emerges.
Neurons are complex. It's arrogant to think we fully understand them.
I think some of the pushback you receive is because you're calling it a mistake when you yourself just admitted that you can't know if it's a mistake because of our incomplete knowledge of neurology.
On the other side of this, there are straightforward arguments that there is some deep connection here. Since LLMs infer statistical relationships of languages produced by human brains, they are in a real sense building a statistical model of how the human brain processes language. It could be the case that this is merely an approximation of some kind, but it could also be exactly how it works.
I think we understand very well that a statistical model can accurately predict the behaviour of a system and still have nothing to do with how the system operates internally.
In a sense, that's the big advantage of building statistical models: you don't need to consider the internals of the system you're modelling.
I agree that's true of some statistical models, I don't think it's true of Bayesian models. Solomonoff Induction has shown that Bayesian inference will reproduce any underlying computable function given a suitable prior.
And this should be obvious given this simple sketch of the argument: classical logic is just Bayesian inference with all probabilities pinned to 0 and 1, and a model of a system in classical logic is a model of how it operates. Therefore you just need to run Bayes' rule long enough for the probabilities to converge.
Many papers have shown that LLMs perform Bayesian inference.
I've heard this before and it's an argument that perhaps makes sense from a Bayesian point of view, but makes no sense from a logic point of view.
To clarify, here's some relevant background on classical logic and probabilities. I will take First Order Logic (FOL) as an example of a classical logic, but the same observations hold true for e.g. propositional logic. In FOL semantics, an interpretation assigns truth values in {true, false} to the atomic formulae ("atoms") of a predicate. In Bayesian inference, a distribution assigns probabilities to the outcomes of an event, such that the sum of probabilities of all outcomes of an event is 1.
Now, for "classical logic [to be] just Bayesian inference with all probabilities pinned to 0 and 1", it must mean that, given an interpretation I, all atoms true under I are assigned the probability "1" and all atoms false under I are assigned the probability "0". This means that the sum of probabilities of all atoms is higher than 1 (it is n times 1, where n is the cardinality of the set of atoms true under I). This makes for a non-Bayesian distribution [*].
So your simple argument sketch doesn't work. Find a better one.
______________
[*] As an aside, all this means that we assume outcomes to be analogous to atoms, events to predicates and distributions to interpretations. There may be a better mapping between concepts and some have been proposed, e.g. that assign probabilities to interpretations rather than atoms, but under FOL semantics the only thing that takes a truth value is atoms, and formulae formed of atoms, and so whatever framework comes out of such transformations has nothing to do with classical logic anymore.
Btw, you can also try to assign different probabilities to atoms than 0 and 1, but then you're outside classical logic again. The fact of the matter is that logic and probabilities are incompatible and any attempt to reconcile the two is doomed to fail. You can speak the language of certainty, or the language of uncertainty, but not both at once.
Btw, when we talk of "functions" we mean a mapping between elements of two or more sets. A "system" is not a function. I don't know a formal definition of a system, but if you're talking about whatever the human brain is doing that's likely to be more easily modelled as a Turing machine.
And a Turing machine is not, in itself, a function. It is a computational device that can calculate the result of some set of calculable functions. You can model a Turing machine as a function, e.g. by mapping some set of programs to the set of inputs and outputs of those programs, or something scary like that, but once again this mapping will tell you nothing about the internal operation of any specific Turing machine.
If, by any chance, you think of the brain as a universal Turing machine (UTM), then I'd really like to see the function you come up with to represent its behaviour; and let's see how much this function tells you about the internal structure of the brain-UTM.
People in my research community have grappled with these questions for a long time and tried to put together the pieces of the puzzle: relations, functions, algorithms, programs, logic, probabilities, statistics, learning, reasoning... the human mind. They have consistently failed because the pieces of the puzzle don't fit.
I think in modern machine learning people have simply given up on trying to make any coherent image out of the puzzle pieces and just bashed them until they yielded and are now wedged together in incongruous patterns that sort of show some kind of image, there. But the incongruities are still felt, like nails on a blackboard.
Re: Bayesian inference and logic
To clarify, I was informally referring to the standard correspondence with propositional logic [1], but generalizations of this to include quantifiers also seem possible [2].
That said, I agree that my simple sketch is not a good justification for Solomonoff Induction, so a poor attempt at a summary on my part.
Re: A "system" is not a function. I don't know a formal definition of a system, but if you're talking about whatever the human brain is doing that's likely to be more easily modelled as a Turing machine.
If you agree that a system can be modelled as a function, then there's a formal equivalence. That's what "is" means to me.
Certainly you can insist on a definition for "is" that's more like a small-step operational semantics to more faithfully model internals, but for the purposes of this discussion, I don't think the internal details matter if the observable inputs and outputs are the same. If a human brain "is" equivalent to some Turing machine, then I can also say that it "is" equivalent to some term in the lambda calculus.
Re: If, by any chance, you think of the brain as a universal Turing machine (UTM)
The brain can be described by a finite state automaton due to the Bekenstein Bound. So it's not a universal Turing machine, strictly speaking. Certainly it would be interesting to know how the internals of the brain work in minute detail, but that's ambitious.
Re: They have consistently failed because the pieces of the puzzle don't fit.
I'm curious specifically what doesn't fit. I understand there's tension between logic and probability, and learning and human-like reasoning are obviously still burgeoning fields, but the rest seem fairly well explored with plenty of correspondences.
Re: I think in modern machine learning people have simply given up on trying to make any coherent image out of the puzzle pieces and just bashed them until they yielded and are now wedged together in incongruous patterns that sort of show some kind of image, there.
Which is kinda what you'd expect from evolution by natural selection.
[1] https://en.wikipedia.org/wiki/Bayes%27_theorem#Correspondenc...
[2] https://www.lesswrong.com/posts/W8YscokXMiDnLKJ96/bayesian-i...
The order of logic doesn't matter. That "standard" mapping of probabilities to logic doesn't work with classical logical semantics and with the common assumption that all probabilities sum to 1. I pointed out why in my previous comment: mapping true to 1 and false to 0 makes for too many 1's to fit in the open interval (0,1). The wikipedia article (which lacks relevant references) and the LessWrong post do not stop to address this foundational inconsistency and instead plow on with building on top of it, but they're only building a house of cards: one cannot draw safe inferences from inconsistent assumptions.
As I say, I get the impression that many such attempts to marry logic and probabilities come from people who understand probabilities, but not logic. They might be considered "standard" in communities that simply wish to do away with logic and only have to think about probabilities, but they are a cop-out supported by sloppy thinking.
On the other hand, there are a number of frameworks that avoid the cardinal sin of mapping true and false to 1 and 0, which they do by eschewing classical semantcis, for example Łukasiewicz logic:
https://en.wikipedia.org/wiki/%C5%81ukasiewicz_logic
These are the obvious choice of logical semantics if one wishes to combine probability with logic, but somehow I rarely see them used. Possibly because they are not as well-known as propositional logic.
In general, I propose to anyone who is interested in the subject to get at least a functional understanding of classical logic, if not as good an understanding as they have of probabilities and statistics.
>> I'm curious specifically what doesn't fit. I understand there's tension between logic and probability, and learning and human-like reasoning are obviously still burgeoning fields, but the rest seem fairly well explored with plenty of correspondences.
What I mean by "the pieces of the puzzle don't fit" is that the various abstractions we have to describe computation don't work when we try to apply them to human thinking. All the formal systems we have are missing something. Trying to work outside formal systems of course leads away from the realm of science and into the world of wild speculation and wishful thinking.
>> Which is kinda what you'd expect from evolution by natural selection.
I don't follow. Evolution by natural selection should be expected to make us lose all hope of building consistent theories?
What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization, https://arxiv.org/abs/2305.19420
An Explanation of In-context Learning as Implicit Bayesian Inference, https://arxiv.org/abs/2111.02080
A Latent Space Theory for Emergent Abilities in Large Language Models, https://arxiv.org/abs/2304.09960
I'm not the one making the affirmative claim absent evidence. I'm the one pointing out the lack of evidence for the claim.
If someone says "there are aliens on the far side of the moon," it's not a mistake to say "there is no evidence for your extraordinary claim".
> On the other side of this, there are straightforward arguments that there is some deep connection here.
No, there isn't.
Humans can perform math. Computers can perform math. No one would claim that's evidence computers think like humans or vice versa.
> they are in a real sense building a statistical model of how the human brain processes language.
And there you go, making exactly the kind of claim I'm talking about.
I'm going to make this very clear: there is absolutely no evidence that supports this claim. Period.
But we would very sensibly claim that computers can think like humans when suitably programmed, and humans can compute like computers. And the claim here is that learning the relationships between words is an understanding of language, and natural language reflects certain kinds of human cognition, and mimicking that output from the same input is mimicking that cognition.
> I'm going to make this very clear: there is absolutely no evidence that supports this claim. Period.
Again, that's incorrect. In what other science could you produce a model that nearly 100% accurately reproduces what the system being modelled would generate, and people would insist on saying that that doesn't really model the operation of that system? Inconsistent standards of evidence IMO.
In any case, There have been a few studies demonstrating strong correlations in activation patterns between the human brain and neural networks. These are correlations, but correlations are evidence.
Furthermore, I think you're failing to understand the argument. Human languages were invented by humans. They are necessarily suited to the human mind, reflecting some fundamental structure and operation of the human brain.
It would be a fairly dramatic coincidence if other, random formal systems were well suited to reproducing natural language. In fact, the most obvious inference is that LLMs are likely inferring semantic models that encapsulate how humans categorize and think, which is why LLMs can translate text between human languages. This would not be possible if languages did not have a common underlying semantic structure.
As far as I'm concerned the problem with claims of great accuracy (100%? Surely you jest?) is that they go above and beyond the evidence provided by modelling. For example, take Convolutional Neural Nets used to train classifiers for objects in images. It is easy to agree that such systems model the human labelling of sets of pixels in digital images, but then this observation is taken as evidence that the systems in question are doing something more than that; namely, that they are somehow modelling human perception.
You can't just take any evidence you have and use it to support any hypothesis you like, and then call that "science". That's not science, it's wishful thinking. If you want machine learning to be scientific you have to be rigorous. But I'm sure if you start talking about scientific rigour in ICML and NeurIPs people will just laugh at you. "Here", they'll say, "we got systems worth millions, what's rigour got to do with anything?".
And they'll be right. Scientific rigour has never made anyone any millions.
The system ostensibly being modelled here is our ability or produce natural language, so the 100% accuracy referred to it's ability to produce syntactically and grammatically correct natural language sentences. This was not a claim that LLMs produce make accurate factual statements with near 100% accuracy. You have to specifically instruct or lead an LLM into violating syntactic and grammatical correctness.
> but then this observation is taken as evidence that the systems in question are doing something more than that; namely, that they are somehow modelling human perception
I think in the case of language this is more likely than not, because language was an invention of the human mind. Therefore the structure of natural language reveals some fundamental properties of the human brain.
Whether this also applies to vision is much less clear, not least because vision models are trained on 2D images and humans are embedded in a 3D space, and the contents of images are not products of the human brain the way language is. There is just no direct parity of reasoning here.
Edit: in some sense, cameras were designed to work well with human perception otherwise they'd have no utility to humans, so there is something about human perception that's revealed, but what's revealed is considerably weaker and more indirect than the natural language case.
You wrote a lot in your other replies and I will follow-up when I have access to more than just a phone and so can provide a more detailed argument and citations.
I see this as overreach also. We have no idea what are the limits of human language ability, and cannot even say how much of human language LLMs can represent. It gets harder and harder to evaluate LLMs the larger they get because we have no better models of language, but that still doesn't tell us how good LLMs are as models of language, it just tells us earlier models are worse. If we still want to be scientific about it, at best what we could say is that a LLM (or any language model) models its training corpus, with some accuracy. That it models our ability to produce natural language? How can we know that, if we don't know what are the limits of that ability? Indeed, a major problem in machine learning research right now is the lack of good tests of language models' abilities.
Sorry for writing too much in the other sub-thread. I'm looking forward to your comments.
> there is absolutely no evidence that supports this claim. Period
The evidence is that it understands and can respond with human language which arose from purely biological processes. So until we understand how this happens we can’t say for sure that these things are not statistical models of part of the human brain. Maybe these models are picking up on Chomsky’s universal grammar or maybe they are emulating brains. We just don’t know.
It might not be strong evidence, it might be indirect, but it is not a total and irrefutable absence of evidence.
If you don't understand how this isn't evidence, I honestly don't know what to do.
> So until we understand how this happens we can’t say for sure that these things are not statistical models of part of the human brain.
I never said that.
I said you can't affirmatively say they are, and further, that because we don't understand how the human brain works, the fact that we can make computers perform tasks that humans can is not evidence in favour of the idea that those computers are in some way modelling the way the human brain actually works. And that is the claim I've seen many people make. In fact that's the claim you just made.
> We just don’t know.
On that we agree.
Well, human mathematical thinking, including logical thinking, also arose from purely biological processes (self-evidently) and yet we have machines that can reproduce all that: digital computers. Perhaps we should consider computers already artificial intelligence?
I actually think that yes, totally, 100% we should. But that makes for a definition of artificial intelligence that will disappoint most people who hope for Star Trek like computer-friends.
https://www.nature.com/articles/s41562-022-01516-2
(Evidence of a predictive coding hierarchy in the human brain listening to speech)
i view this as the newtonian mechanics vs actual reality debate. the former may not be a very accurate model of the latter, but it is very useful and it would be wrong to say there is no similarity between them.
There's always been the two kinds of ML research, the "experimental math" and the "understanding the math" kind, with a degree of overlap.