Models are weights for matrix operations, they are determenistics.
They go through this rigamarole because a little bit of randomness gives better results from a Turing test kind of perspective.
Not to be snarky or dismissive, I mean this genuinely: ask an LLM about it. I currently have a headache so I'm not up to explaining the technical details, but they are interesting and worth reading about.
Issues include accumulated floating point errors happening in different orders due to distributed and parallel computation, CUDA kernels that deliberately sacrifice determinism for speed, and several other such issues.
I think you likely right, that some parts of stack could induce some marginal float point error, but converged model can mitigate it, and on some principal set of knowledge can give deterministic result with high probability.
Which leads me to believe if you give this task to Anthropic, who has very strong incentive, they will build such benchmark, and then can tell that benchmark gives correct answer with 99.9% probability and it will be enough to drag someone to court.
However, running in production at any sort of scale often involves multiple machines and multiple GPUs, and at that point, determinism can be difficult to achieve.