(to be clear: I submitted but not the author of the library myself)
> OpenAI models are non-deterministic, meaning that identical inputs can yield different outputs. Setting temperature to 0 will make the outputs mostly deterministic, but a small amount of variability may remain.
I guess I was imagining that if temperature was 0, and the model was not being continuously trained, the weights wouldn’t change, and the output would be deterministic.
Is this a feature of LLMs more generally or has OpenAI more specifically introduced some other degree of randomness in their models?
Setting temperature to 0 makes the output deterministic, though in my experiments it's still highly sensitive to the inputs. What I mean by that is while yes, for the exact same input you get the exact same output, it's also true that you can change one or two words (that may not change the meaning in any way) and get a different output.
So for example, the input:
Bears like to eat ________
GPT may effectively respond with Honey (33% likelihood that honey is the word that follows the statement) and Humans (30% likelihood that humans is the word that follows this statement). GPT is just estimating what word follows next in the sequence based on all it's training data.
With temperature = 0, GPT will always choose "Honey" in the above example.
With temperature != 0, GPT will add some randomness and would occasionally say "Bears like to eat Humans" in the above example.
Strangely a bit of randomness seems to be like adding salt to dinner - just a little bit makes the output taste better for some reason.