LLM Inference Determinism
Quick Answer
LLM inference determinism is the property that a fixed (prompt, model, decoding config) tuple returns the same output tokens on every call. It is a layered engineering property, not a default: greedy decoding and a fixed seed only control the sampling layer, while batch shape, GPU model, kernel selection, and library versions still perturb floating-point results enough to change the argmax token.
LLM Inference Determinism
LLM inference determinism is the property that a fixed (prompt, model, decoding config) tuple returns the same output tokens on every call. It sits below reproducibility, which also covers data, prompts, and orchestration. Determinism is layered: sampling controls (greedy decoding, temperature zero, fixed seed) pin only one layer, while batch shape, GPU model, kernel choice, and library versions shape the underlying floating-point arithmetic. Small perturbations can flip the argmax token, and in reasoning models an early divergence cascades through the chain of thought. Even with every layer pinned, published work shows bit-for-bit equality is not guaranteed.
Every lever requires owning the inference stack, so behind a hosted API determinism is unavailable rather than merely expensive. The practical substitute is to freeze retrieved inputs into an append-only record and replay the deterministic downstream stages against it. That reproduces the reasoning rather than the tokens — usually what actually matters.
See also
- What building a model-agnostic research pipeline actually took — why an evidence ledger replaced seed pinning in a hosted-API pipeline.