Yep. That's why predicting the future of a system you're a part of is sometimes literally impossible; it boils, fundamentally, to the fact that logical negation doesn't have a fixed point.
Like, if you output a probability distribution among n options, and then there is a continuous map from the probability distribution you describe in your output, to another probability distribution over those options…
Err, hm, maybe you need a stronger hypothesis on the continuous map? If it is contractive then it will definitely have a fixed point. I don’t remember the hypothesis needed.
Also, consider the game of hunter-and-prey from my reply to the sibling comment. Again, the prediction that the prey will make some choice is arguably not a prediction at all.
Now being serious, LLMs are nondeterministic loosy compression algorithm which return "what a plausible answer to that question would be". They have no world model, no truth verification and formally will be excellent while having no verifiable content.
"There is a book about a boy in a magical school, with friends, who fights a big bad and its called ..."
should have Harry Potter as the highest possible response. But that is not the answer based on a truth model, just a statistical average. You could have been looking for Earthsea series. Or it could have the heat turned up and decide to go for a hallucinated answer over the statistically most likely token.
If you want a verifiable answer your best options would be to have the heat turned to 0, but that means it cannot create new responses. Or if you have heat, to ask 4 times and sample the avg. But at that point what is the advantage over just googling it?
And one shooting your answer and hoping its the bell curve answer is going to have decent results but its also proveably going to fail in a non insignificant number of cases
I don't see how that helps? If you just want the same answer multiple times in a row, you could just write down the seed of your pseudorandom number generator?
> And one shooting your answer and hoping its the bell curve answer is going to have decent results but its also proveably going to fail in a non insignificant number of cases
Setting the temperature to 0 doesn't guarantee you get a good answer either.
Now, if the hunter can accurately predict the prey's choice, they obviously simply need to select that same choice to always win. The similar logic works for the prey; but this means they can't both be perfect predictors of the future — if the hunter predicts that the prey will choose T, they will choose T as well, but that means the the prey will predict the hunter choosing T and will choose F instead, so the hunter has actually to predict that the prey will choose F, and so they will choose F as well, but that means the the prey will predict the hunter choosing F and will choose T instead...