Hacker News Reader

Comment by etaioinshrdlu | Hacker News Reader

The backend OpenAI APIs are not deterministic even with temperature 0, and they might upgrade the model weights/tuning behind your back too? (Not sure about the upgrade, they might just put out a new model id param...)

View on HN

It's very strange that they don't guarantee determinism. We trained a model at Codeium that does have determinism at inference time - it's not super hard if you don't just use random seeds everywhere.

This makes it a lot harder to rely on OpenAI's API if you are doing any sort of caching / evaluation workloads.

Not promising determinism gives them a lot more implementation wiggle room to update / change things.

My guess is that promising it right now is not a priority. It would be a great "pro" or "enterprise" type feature tho.

Pytorch/CUDA is apparently sometimes nondeterministic https://twitter.com/taliaringer/status/1511411984398487564?s...

Interesting, I guess we implemented this op properly :)

It is strange that cub does not guarantee determinism here - it's not too hard an op to implement in CUDA.

They seem pretty consistent with temperature 0. Maybe not 100% but very close.

“Pretty consistent” and “deterministic” are not at all the same

It's close but not fully deterministic. I remember seeing a theory that if the system is considering multiple possible completions that have the same rounded score percentage, then its choice between them is nondeterministic.

It's likely because GPU calculations are non-deterministic and small differences in floating point numbers could lead to different outcomes (either in the way you described or somewhere deeper in the model)

> GPU calculations are non-deterministic

Tensorflow is non-deterministic for some operations due to thread scheduling. PyTorch doesn’t have this issue.

Some of the underlying CuDNN algorithms have nondeterministic implementations, which applies to PyTorch as well. See https://pytorch.org/docs/stable/notes/randomness.html

normally you can choose the model version so that's not a concern.

Reply on news.ycombinator.com