LLMs are the the best conmen. If they were humans people would trust them their life savings for sure.
“How sure are you on /20”
If it says yes I am sure, and other LLMs confirms the same way, then you can be fairly confident that it is a very good candidate answer.
If it says he is not sure, he is probably just agreeing with you, and better double-check by asking “is there other solutions ? What is the worst idea ?” Etc, to force it through thinking and context.
It is cross-validation, and you can even cross-validate by searching on the internet.
Though, 100% they say what you them want to say.
Except on religion and immigration, and some other topics where it will push its own opinion.
For example, if it claims A > B, then it shouldn't claim B > A in a fresh chat for comparisons.
In general, you shouldn't get A and not A, and you should expect A or Not A.
If it can go from prompt -> result, assuming it's invertible, then result -> prompt should also partially work. An example of this is translation.
The results of some mathematical solutions should go back and solve the original equations. Ex. The derivative of an antiderivative should give you back the original.
For example the other day i asked ChatGPT about a problem i had with some generated code that didn't compile. It then told me about a setting in the generator (nswag), that didn't exist. I told it that this setting does not exist and it said something like: "Sorry, my bad, try the following" and then kept inventing this setting with slightly different names and values over and over again. There are similar settings that exist, so it just hallucinated a tiny bit of text inside all the snippets that it learned from.
This is also not the first time this happened, most of the times i tried using AI for help with things, it just made up some nonsense and wasted my time with it.
Artificial colors and artificial flavors and artificial intelligence all have something in common - they are not as good as the real thing, and probably never will be.