Hacker News Reader

Comment by mmastrac | Hacker News Reader

If the results were stable, reproducable and somehow memoizable, this would be actually insanely useful. Perhaps it could modify a cache file somewhere with the generated Python code, and that could be committed.

View on HN

The backend OpenAI APIs are not deterministic even with temperature 0, and they might upgrade the model weights/tuning behind your back too? (Not sure about the upgrade, they might just put out a new model id param...)

normally you can choose the model version so that's not a concern.

They seem pretty consistent with temperature 0. Maybe not 100% but very close.

“Pretty consistent” and “deterministic” are not at all the same

It's close but not fully deterministic. I remember seeing a theory that if the system is considering multiple possible completions that have the same rounded score percentage, then its choice between them is nondeterministic.

It's likely because GPU calculations are non-deterministic and small differences in floating point numbers could lead to different outcomes (either in the way you described or somewhere deeper in the model)

> GPU calculations are non-deterministic

Tensorflow is non-deterministic for some operations due to thread scheduling. PyTorch doesn’t have this issue.

Some of the underlying CuDNN algorithms have nondeterministic implementations, which applies to PyTorch as well. See https://pytorch.org/docs/stable/notes/randomness.html

It's very strange that they don't guarantee determinism. We trained a model at Codeium that does have determinism at inference time - it's not super hard if you don't just use random seeds everywhere.

This makes it a lot harder to rely on OpenAI's API if you are doing any sort of caching / evaluation workloads.

Not promising determinism gives them a lot more implementation wiggle room to update / change things.

My guess is that promising it right now is not a priority. It would be a great "pro" or "enterprise" type feature tho.

Pytorch/CUDA is apparently sometimes nondeterministic https://twitter.com/taliaringer/status/1511411984398487564?s...

Interesting, I guess we implemented this op properly :)

It is strange that cub does not guarantee determinism here - it's not too hard an op to implement in CUDA.

Looks like Dill [0] would be a good option for serialising the generated code. The built in Python Pickle doesn't support pickling functions properly, but Dill extends it to enable this sort of functionality.

0: https://pypi.org/project/dill/

Per the README (and the source code), requests to GPT are cached. I know that doesn't solve the stability problem, since the cache is keyed on the exact prompt among other parameters, but it's something at least.

The source code shows that it's using pickling to store OpenAI responses on the local filesystem. See here:

https://github.com/ianb/infinite-ai-array/blob/main/iaia/gpt...

and here:

https://github.com/ianb/infinite-ai-array/blob/main/iaia/gpt...

It really should be saving them to JSON instead of Pickle, but I gave up trying to figure out how to properly rehydrate the openai Completion objects.

If it was JSON then it wouldn't be crazy to add it to your repository. (I guess Pickle is stable enough in practice, but it would offend my personal sensibilities to check them in.)

What is wrong with Pickle?

Arbitrary code execution is a big one.

I'm aware, but it is not an issue if you only use it with your own data. You can even cryptographically sign it.

Besides security it's pretty opaque, and unnecessarily so. The data all comes over JSON so there's no non-JSONable types involved. It would be nice to be able to run tools like cost summarizing over the JSON without having to load up any other code.

In what manner do you reckon would it be insanely useful?

I'm no python expert but it looks like this does happen in memory: https://github.com/ianb/infinite-ai-array/blob/main/iaia/mag...

I can't imagine it's too hard to serialize the `existing` python dict so subsequent runs are deterministic

It is also caching requests on-disk. See for instance https://github.com/ianb/infinite-ai-array/blob/main/iaia/gpt...

The problem* is that if a prompt changes slightly then it won't be a cache hit.

* okay, one problem.

Reply on news.ycombinator.com