Hacker News Reader

Comment by haolez | Hacker News Reader

Wait. Is temperature=0 "pretty deterministic", or is it deterministic?

In the strict definition, it's deterministic: you get the same response for the same prompt, each time (given the exact same model).

But the prompt is parameterised. The bulk of the prompt is requesting a list of guests and speakers to be extracted, and the episode synopsis is appended as the "parameter". And I've noticed that the variation of the parameter changes what the overall prompt returns... so it might start being less reliable at responding with valid JSON, for example.

So it's instance-deterministic but, across a range of parameters, class-fuzzy, if that makes sense?

Temperature=0 is not perfectly deterministic for the OpenAI API.

It's mentioned briefly in the OpenAI text completion guide: https://platform.openai.com/docs/guides/completion/introduct...

If you have two possible tokens with probability 40% and 30%, you'll always get the 40% token at T=0. But if you have two possible tokens at 40% and 39.99%, you may get the 39.99% token on occasion, even if at T=0. (Numbers illustrative.)

Where does the inherent randomness come from?

It's injected into the model, you can deviate from no randomness to get "creative".

Isn't the temperature the "injected" randomness? The way I imagine it would be somewhere in the model you would do

input += (Math.random - 0.5) * Coefficient * temperature

so setting temperature to 0 would mean no randomness. On thinking further about why there is inherent randomness I believe it is from a lack of associativity in floating point operations. They obviously do A LOT of parallel floating point operations.

On a related note, has anyone noticed the growing use of "pretty" as a hedge against ever being wrong? Also, "not", as in "not the fastest" or "not the hottest".

It could be 46C/115F outside and you say "wow, today is unbearably hot", to which someone retorts "nah, it's fine. It's not the hottest day". That's pretty good hedging. You can make infinite technically correct statements this way without ever saying anything meaningful

> Also, "not", as in "not the fastest" or "not the hottest".

Mid twenties Hiberno-English speaker here and that's always been a fairly common form of hedging, and I've not noticed an increase.

fairly, lol

It’s my understanding that 0 is completely deterministic, except for when the model is updated, which does happen occasionally.

@haolez I just tested w/ davinci-003 w/ temperature to 0

Prompt: https://imgur.com/YtQ4fbf --

Reveal the question marks in an interesting way:

The dog goes ????????????? --

With temperature == 0, it consistently ("pretty deterministically"?) generated "woof!"

ex. The dog goes woof!

Their docs say (somewhat recently updated, I think) that even 0 is not perfectly deterministic in all cases (though it's very close). Some people had previously observed this and speculated that it was some kind of floating point roundoff issue, when two outputs have almost identical scores.

GPU's often do floating point computations in nondeterministic order for performance reasons, so you would get small differences even for exactly the same order. It is probably that.

Well technically everything is deterministic.

Reply on news.ycombinator.com