It looks like all current models suffer from an incurable case of Dunning–Kruger effect cognitive bias.
All are at the peak of Mount Stupid.
But they can also only do negation through exhaustion, known unknowns, future unknowns, etc...
That is the pain of the Entscheidungsproblem.
Even in Presburger arithmetic, Natural numbers will addition and equality, which is decidable, still has a double factorial time complexity to prove. That is worse than factorial time for those who've not dealt with it.
Add in multiplication then you are undecidable.
Even if you decided to use the dag like structure of transformers, causality is very very hard.
https://arxiv.org/abs/1412.3076
LLMs only have cheap access to their model probables which aren't ground truth.
So while asking for a pizza recipe could be called out as a potential joke if add a topping that wasn't in its training set, through exhaustion, It can't know when it is wrong in the general case.
That was an intentional choice with statistical learning and why it was called PAC (probably approximately correct) learning.
That was actually a cause of a great rift with the Symbolic camp in the past.
PAC learning is practically computable in far more cases and even the people who work in automated theorem proving don't try to prove no-instances in the general case.
There are lots of useful things we can do in BPP (bounded probabilistically polynomial time) and with random walks.
But unless there are major advancements in math and logic, transformers will have limits.
The parameters don't store any information about what inputs were seen in the training data (vs being interpolated) or how accurate the predictions were for those specific inputs.
And even if they did, the training data was usually gathered voraciously, without much preference for quality reasoning.
Multiple sub-networks detect the same pattern in different ways, and confidence is the percent of those sub-networks that fire for a particular instance.
There's a ton of overlap and redundancy with so many weights, so there are lots of ways this could work
I guess that is a slight variation of the sibling (@habitue's) answer; both are checks against external material.
I wonder if best resources could be catalogued as the corpus is processed, giving a document vector space to select resources for such 'sense' checking.
https://en.m.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effec...