Hacker News Reader

Comment by PeterStuer | Hacker News Reader

"these are non-deterministic machines"

Only if you choose so by allowing some degree of randomness with the temperature setting.

They are usually nondeterministic even at temperature 0 - due to things like parallelism and floating point rounding errors.

This is dependent on configuration, you can get repeatable results if you need them. I know at least llama.cpp and vllm v0 are deterministic for a given version and backend, and vllm v1 is deterministic if you disable multiprocessing.

floating point rounding errors are still deterministic. Parallelism dynamics can impact results, but those are not specific to LLM's.

Here's something that isn't deterministic:

   a = 0.1, b = 0.2, c = 0.3
   a * (b * c) = 0.006
   (a * b) * c = 0.006000000000000001

If you are running these operations in parallel you can't guarantee which of those orders the operations will complete in.

When you're running models on a GPU (or any other architecture that runs a whole bunch of matrix operations in parallel) you can't guarantee the order of the operations.

The order of completion doesn't necessarily influence the overall result of a parallelized computation. This depends on how the results are aggregated. For example for reducing floating point error in calculating a sum of floating point numbers, you could have a sorting step before calculating the sum and then start summing up from the lowest values to the higher ones. Then it doesn't matter at all which of the values is calculated first, because you need them all anyway, to sort them and once they are sorted, the result will always be the same, given same input values.

So you can see, completion time is a completely orthogonal issue, or can be made one.

And even libraries like tensorflow can be made to give reproducible results, when setting the corresponding seeds for the underlying libraries. Have done that myself, speaking from experience in a machine learning setting.

In the strict sense, sure, but the point is they depend not only on the seed but on seemingly minor variations in the prompt.

This is what irks me so often when reading these comments. This is just software inside a ordinary computer, it always does the same with the same input, which includes hidden and global state. Stating that they are "non-deterministic machines" sounds like throwing the towel and thinking "it's magic!". I am not even sure what people want to actually express, when they make these false statements.

If one wants to make something give the same answers every time, one needs to control all the variables of input. This is like any other software including other machine learning algorithms.

Reply on news.ycombinator.com