Hacker News Reader

LLMs are, in theory, deterministic. Sampling is not intrinsic to LLMs.

Greedy decoding a single batch in most libraries will give you mostly deterministic outputs. Higher batch sizes can increase variance.

But all of this is down to CUDA and/or kernel implementation issues.

View on HN

It's a more or less solved problem (e.g. DeepSeek has batch-invariant kernels, vLLM also has reproducible outputs etc), with an obvious caveat that you need to lock your hardware setup, model snapshot, and inference stack if you want any useful reproducibility across longer time periods. It's up to specific providers to actually implement it to any extent, of course commercial providers are incentivized to do the opposite.

Reply on news.ycombinator.com