Are llms truly non-deterministic?
My impression was that we inject some randomness intentionally to mimic it, but if you remove that factor you end up with a deterministic chat model.
My impression was that we inject some randomness intentionally to mimic it, but if you remove that factor you end up with a deterministic chat model.
It's also "non-deterministic" in the sense that if you removed all sources of non-determinism and asked "What is 1+1?" and received the answer "2" deterministically, that doesn't guarantee a correct answer for "What is 1+2?". Ie a variation in the input isn't correlated in a logical way with a variation in the output, which is somewhat fatal for computer programs, where the goal is to generalize a problem across a range of inputs.
So the total generation of text from an LLM can be made fully deterministic. The problem for scientists is that we cant do that in the deployed systems...