Since the LLMs are probabalistic is it possible for the inference engines to highlight lower probability content. That might at least make them more "human" rather than being confident and inaccurate.
But another problem is that confidence is a character attribute, not a writer attribute. If you ask an LLM to imitate Richard Feynman it's going to write pretty confidently about physics, despite not knowing as much as him about physics.
This is the equivalent of giving your RPG character high intelligence on the character sheet. Doesn't make you smart!
To make an LLM express confidence consistent with its actual knowledge, it would need to have good self-knowledge and actually use it. So far this doesn't happen automatically. Instead, OpenAI uses reinforcement learning based on what the people at OpenAI think the LLM can do. So that's why it sometimes refuses to answer for some kinds of questions.
That has the most effect on the default character, the "helpful AI assistant". Any other characters you ask for will likely have poorer self-knowledge.
I've read that, mysteriously, more training does make bigger LLM's better calibrated, but the reinforcement learning makes it worse again.
Like "Please reply T if you are confident about the text that you are going to generate, and F if you are not".
But if this could be done by architecting the neural networks, they would perform much better.
We don't fully understand how they work or what their limits are.
It's also worth remembering that this is not just a markov chain as many people seem to think, it doesn't simply remember what words come next in its training set, because statistically if you take a random 10 consecutive words from a piece of text, chances are its never been written before. (also the trained model is much smaller than the size of the dataset, so to simply remember everything it would have to be the worlds best compression algorithm by orders of magnitude) Thats why we need an AI here, to learn the general rules of language so it can respond to chains of words it has never seen before.
The sense that we "dont understand how they work" is that we dont know what the "rules of language" that it has learned are.
This is no more helpful in understanding AI's than is knowing that human brains operate according to the laws of physics is helpful in understanding the human mind.
[1] https://skybrian.substack.com/p/dont-settle-for-a-superficia...
[2] https://skybrian.substack.com/p/ai-chats-are-turn-based-game...