> the actual execution is very much non-deterministic.

Only as a default. Temperature can be turned to 0 and other token selection settings changed from the defaults to create deterministic output from a given prompt.