Can you elaborate?
Is it because the models are inherently not deterministic? But then, will it not get better with time? I mean arent we just at the beginning of the research here?
Determinism isn't the issue; it's that the prompt does not contain enough information to know the correct way to do the thing, even if it did the thing the same way with the same inputs every time.
This is the main reason (and yes, many modern compilers and CPUs carry some non-determinism which actually quite well explained and specified).
> But then, will it not get better with time?
No, it won't. Bacuse that is that is the actual literal limitation/feature of LLMs.
> I mean arent we just at the beginning of the research here?
In general? Yes. With LLMs? We can reasonably say that they will never be deterministic.
There is a way to get a non-determenistic output: first question on a temp 0 local model on a completely new session will give you the same answer. The second answer in that same session will already be different on every session (even if it's the same question).