Hang on - I thought the consensus among ML experts was that language models don’t ‘know’ anything?
The models still capture statistical regularities in the "language" (overwhelmingly that is text corpora) they were optimized for tasks against, and to the extent what went in during optimization really represents facts and relations in the world, the model probably holds representations related to facts and relations in the world.
It probably also holds representations of spurious facts and relations in the world. The optimization on language is not explicitly designed for capturing semantic information and, as far as I know, there's typically no mechanisms built in to measure or report anything about the veracity of the claims. Which makes relying on anything coming out of them a shaky business at best for now of course.
The point of this kind of work is to query the LLM and figure out what it ‘thinks is true’ - which is then presumably useful for deciding how much to rely on its output.
I just find it amusing that people in the ML field are always so adamant that ‘it doesn’t know’ or ‘it’s not thinking’ or ‘it doesn’t understand’, but I don’t see much engagement with what is actually meant by ‘know’, ‘think’ or ‘understand’ in the first place.
Like, you just said that a language model ‘probably… holds representations of spurious facts and relations’.
But… isn’t ‘holding a representation of a fact or relation’ basically what we call ‘knowing’ something?
Exactly. This is why these experts should shut up about claiming these AI models don't know things, or aren't sentient. We don't have any mechanistic explanations for what knowledge is or how it works, or what sentience is or how it works, so all of these claims are just bullshit. At best we have some hand-wavy intuitions about what some properties of knowledge or sentience have (Gettier problem, etc), but that's nowhere near enough to make the definitive claims I've seen.
No. Knowledge is fundamentally and inextricably tied to our experience of reality. The billions of years of evolution that created our perceptual framework has created a bridge to understanding reality machines don’t have.
I’d argue a chicken brain has a far better greater amount of true knowledge than any of these language models.
That does not diminish the utility and sophistication of these language models. But they are not knowledge. They are a disconnected meta map of what we have written down.
We calibrate these language models to knowledge because our language usually corresponds to things we know.
If the internet was mostly populated with sentences like “a druid flew into houston, cats pool into mud vampires wealth umpire onion buy now saxophone”, a language model mapping it all would be useless.
What is your name for the quality that such a language model possesses, that makes it more ‘useful’?
Why is ‘knowledge’ not a good word for that quality?
I think knowledge is in fact a good word for ‘sensical’ language, but the point most people like me are trying to make when talking like this is that language models can’t determine what is ‘sensical’ language.
The amazing thing about these newer models is that they seem to get remarkably close to properly understanding real world concepts just through symbol mapping on massive amounts of data, but that’s highly dependent on having being fed sensical data where that mapping of symbols to reality was already done.
I’m trying to argue that knowledge only really exists in the data feeding step, when we map symbols to reality.
What the language models do is create a map of symbols. But they mix things which are knowledge with things that are not, and cannot determine what about their output and their mapping is actually valid knowledge.
I think most of why this is has to do with the nearly endless amount of embodied physical and historical context that goes into our own sense making. A surprisingly large degree seems to be embedded in language, but there’s a lot more that is not.
EDIT: I rambled, sorry; direct answer to your question about what I’d call the property that makes these language models useful is “relatively high degree of correspondence with reality”. What I consider “knowledge” is directly obtained by something which can interact with reality/directly maps reality to some representation.
That’s actually precisely why I think evolutionarily adapted embodied perceptual systems need to be a part of any true knowledge: reality is so incredibly complex that we don’t know what our perception is or is not doing. We just know it’s doing something reality based because we exist. Perceptual models don’t need to correspond with reality like actual embodied perception (and corresponding cognition)
I don’t buy the argument that the correspondence with objective reality in evolved perceptual systems is zero as is claimed in that interview, simply because we exist. I think that conclusion is an artifact of precisely the same kind of over abstraction and non-correspondence that invalidates language models as knowledge systems. Our existence means our perception corresponds with at least some kind of compatibility with and ability to navigate through reality.
A system which does not have that embodied evolutionary history with millennia of required correspondence with reality, simplified or not, does not have all of the essential embedded context and understanding of complexity that goes into our sense making. Science is about discovering more of that hidden context, and perhaps these language models can help identify patterns we haven’t seen yet that do in fact correspond with reality. But the model can’t test it’s own correspondence.