> The only nondeterminism in LLMs is parallelism-dependency in the way floating point operations are carried out and in the decoding strategy

This assertion is "oddly" similar to the GPT answer "neural network inference determinism" produced:

  Neural network inference is often non-deterministic due to 
  factors like floating-point arithmetic and concurrent 
  execution, which can lead to variations in output even with 
  the same input.

Surely this is but a coincidence.

Regarding your previous statement:

> However, on a technical level, neural network inference truly is inherently deterministic.

This holds for a vanishingly small set of conditions, none of which include randomness, nor when context and transformers are involved, let alone underlying model evolution (thus making model use over time non-deterministic).