It's not a problem when you are aware of it and with some follow up input you can get it mitigated, but often I see that people tend to take the first output of these systems at face value. People should be a bit more critical in that regards.
How do you benchmark something or someone understanding text?
I'm asking because the magic of LLM is the meta level which basically creates a mathematical representation of meaning and most of the time, when i write with an LLM, it feels very understanding to me.
Missing details is shitty and annoying but i have talked to humans and plenty of them do the same thing but actually worse.
I guess at best you can say these models have an ‘understanding’ of language, but their ability to waffle endlessly and eruditely about any well-known topic you can throw at it is just further evidence of this — not that it understands the content.
You can also just quiz it on certain basic definitions. Ask it for examples of objects that don’t exist (graphs or categories with certain properties, etc.). Sometimes it’ll be adamant that its stated example works, but usually it will quickly apologise and admit to being wrong only to give almost exactly the same (broken) argument again.
Another thing you can try is concocting some question that isn’t even syntactically well-formed (i.e. fails even a type check) like ‘is it true that cyclic integer lattices are uniformly bounded below in the Riemann topology?’. I imagine that one is too far out to work, but when I’ve played around I’ve found many such absurd questions ChatGPT was only too happy to answer — with utter nonsense, of course. It’s interesting (and, I think, quite telling) that such systems are seemingly almost completely unable to decline to answer a question. And the reason is that there’s no difference between hallucination and non-hallucination. Internally, it’s exactly the same process. It either knows or doesn’t know — but it doesn’t know that it knows (or doesn’t).
LLMs basically only work on questions that are very similar to, or identical to, questions that have already been widely asked and answered online or in books… hence their lack of utility in mathematical research, or even in calculating one’s taxes, or whatever.
I could provide some more literal examples, but I’d have to go and try some and pick the ones that work, and even then they might not work on your end because of the pseudorandomness and the fact that the model keeps getting updated and patched. It’s better to just play around on your own based on the ideas I’ve given.
The moral is to use LLMs as a powerful way of finding information, but don’t trust anything it says. Use it to find better sources more quickly than you’d be able to via a search engine.
If you for instance threatened to shoot a human if it refused a request or admitted it didn't know something, they might answer in a very similar fashion.
Interestingly enough, tx to a talk from a brain researcher i understood that there are two major brain modes we run: Either the i just observe and do things i observed (were it doesnt' matter that you are gay or black but still vote for trump) and the logical mind were you see a conflict in stuff like this.
LLMs are interestingly enough somewere inbetween.