It's a language model, so it's good at generating language. Any congruency with knowledge of what the language expresses is purely accidental side effect.
Humans who can't do math might be a pure language models though.
It certainly "knows" some stuff. However it can't reason about it.
There have been some articles on HN recently on how accurately LMs build world models.
I think the key bit here is that we can influence its associations by inputting the prompt, changing the model that is presented.
I'm waay ahead of myself here, but the thought is interesting, and it will likely remain an open question for at least a few months/years.
Which is actually a novel capability and arises because the network does reinforcement learning over its own context window. It's a strength, not a weakness. Humans can do the same thing. ("Assume that X...")
> It just randomly landed on correct thing first just because it seen it more often in the input data.
Isn't that just a description of learning?
It's true that the network has no idea what is "true". But it's not like we do either, all we do is learning from correlations. We're just better at it.
Put otherwise, GPT doesn't interact with reality directly, but mediated through our language. But we also don't experience reality directly, so it's hard to see how much of a hindrance that would pose. Hell, much of our model of the world is just as rooted in language as GPT's. The part of my model of reality of which I have direct sensory experience is tiny.
Yes, but the human language is garbage. So while passing throught that filter without maintaining the highest degree of precision barely any of the actual reality remains. Truth sounds exactly the same as a lie.
The actual language that decently captures reality is math. Nothing less is even barely sufficient to produce almost anything but nonsense. Nonsense nicely sounding to human ears but still nonsense.