Hacker News Reader

Comment by jpgvm | Hacker News Reader

They can be deterministic. We did this at Groq, if you sent a request with exactly the same input token, seed and temperature value you would get precisely the same result every time.

This is harder to do on other architectures that themselves aren't fully deterministic though.

View on HN

Was the temperature 0? Cause unless I don't understand it right, any non-zero temperature implies probabilistic next token prediction.

You did mention, seed, which I haven't seen available anywhere else (in my limited experience) and it can explain a reproducible inference result, but I feel like this matches the "letter of the law" meaning of deterministic rather than the "spirit of the law".

If you send the seed along, you can have a non-zero temperature: when it's time to select tokens, a PRNG will be seeded and used for the selection.

Using a PRNG and a seed, you can have paeudo-randomness and determinism. The pseudo-randomness is the key piece, it’s random to the extent that it cannot be distinguished from true randomness in polynomial time classically.

Perhaps they just used the exact same random seed everywhere?

Do you mean that you get the exact same result (as word for word) because from Groq's documentation, it says that determinism is best effort, and also it's not guaranteed across model versions : https://console.groq.com/docs/prompting?_sm_vck=tMW2S7k17245...

My understanding is that it's best effort because even if you control the seed/temperature, the model version and hardware specs you still have floating point math that will introduce rounding differences

an article explaining this in detail : https://news.ycombinator.com/item?id=45200925

> harder to do on other architectures that themselves aren't fully deterministic though

The TLDR here is that Groq LPUs are completely deterministic hardware, including for floating point.

Disclaimer: Formerly of Groq, now of NVIDIA, still working on LPUs.

Reply on news.ycombinator.com