AskpertWhy does the same prompt give different answers at temperature 0?
Temperature 0 buys greedy decoding, not reproducibility. The cause is not the GPU race condition everyone cites, and it changes how you read an eval score, a regression, and a bug report.
On this page
Terms, definedthe jargon, decoded
- Greedy decoding
- Selecting the single highest-scoring next token at every step, rather than sampling from the distribution. This is what temperature 0 does.
- Logits
- The raw scores a model assigns to every possible next token before they are turned into probabilities.
- Batch invariance
- The property that a request produces the same numbers regardless of how many other requests were processed alongside it. Most inference kernels do not have it.
- Reduction
- A kernel operation that combines many values into one, such as summing a row. Its order affects the rounded result because floating-point addition is not associative.
- vLLM
- A widely used open-source engine for serving language models.
The usual explanation is wrong, and it is wrong in a way that matters.
Engineers attribute variation at temperature 0 to random sampling, floating-point non-associativity, or GPU race conditions. Sampling is not the explanation once decoding is greedy. The decisive condition is whether the inference implementation is batch invariant. Get that wrong and you will spend a day diffing prompts to explain something that happened in the scheduler.
What does temperature 0 actually guarantee?
Temperature 0 makes decoding greedy: at each step the decoder takes the token with the highest computed probability. It removes intentional sampling from that choice, and nothing else. It does not guarantee that the computed probabilities, the logits, or the winning token stay identical between runs.
If the numbers move enough for two close candidates to swap rank, greedy decoding picks a different token, and every token after that is conditioned on the new one. A deterministic decoder can consume nondeterministic logits. That is the whole distinction: deterministic selection is not deterministic computation.
| Property | Temperature 0 | Deterministic inference path |
|---|---|---|
| Token selection | Greedy, by highest computed logit | A fixed computation path and decoding rule |
| Sampling randomness | Removed from the selection step | Controlled by the decoder |
| Batch sensitivity | Not removed | Removed or bounded by the implementation |
| Reproducibility | No general guarantee | Depends on the serving system |
An API can expose temperature 0 while its serving path still permits batch-dependent variation.
Why can the same prompt produce a different answer?
Because server load changes the request's effective batch size, the batch size changes the numerical reduction order inside the kernels, and that changes the logits greedy decoding reads. Your result depends on who else happened to be hitting the same server.
Thinking Machines Lab documented the mechanism in its September 2025 engineering post, Defeating Nondeterminism in LLM Inference. This is a lab engineering write-up rather than a peer-reviewed paper, and it is worth reading in full. Testing Qwen3-235B-Instruct with 1,000 completions at temperature 0, each capped at 1,000 tokens, produced 80 unique outputs. The first divergence landed at token 103, where 992 completions said "Queens, New York" and 8 said "New York City."
The same post rules out the folk explanation. A repeated forward pass on the same hardware gives bitwise-identical output, so GPU concurrency, floating-point non-associativity and atomics are not a sufficient account on their own. The instability shows up when the batch shape changes.
How does batch size change an answer?
Batch size changes the dimensions and execution path of the work a kernel does. A reduction combines many values into one, floating-point addition is not associative, and reordering those additions can change the final rounded value. A tiny numerical difference is enough to flip the ranking of two nearly tied tokens.
Meanwhile a serving system is batching requests together to keep the accelerator busy. The active batch differs between two of your calls because other traffic arrived, finished, or ran to different lengths. Your prompt is constant; the computation around it is not.
The chain is short:
- Other requests alter the active batch.
- The batch shape alters kernel reduction order.
- Reduction order alters floating-point results.
- Altered logits change a greedy token choice.
- The first changed token changes everything after it.
Because of step five, comparing the first divergent token tells you far more than comparing final strings. For an agent, compare the first different tool call or branch.
What does this mean for an evaluation score?
A single evaluation run is a sample of system behavior, not a measurement of reliability. If identical requests can produce different completions, one score conflates task performance with one realized inference path. This is the mechanism underneath the argument that reliability is a distribution, not a score: batch-dependent nondeterminism widens that distribution while your code, prompts, and model identifier all stay put.
So the run count belongs in the result. An evaluation record should say how many times each case ran, which outputs passed, and whether the failures repeated. Without those, a score cannot separate a real regression from execution variance. For an agent the stakes are higher, since one changed token can alter a tool call, a parsed argument, or a branch condition, and send the whole trajectory somewhere else. The same variation has a constructive reading, though: sampled deliberately, it raises the odds a correct answer appears at all, which a verifier can then collect.
Can a regression appear overnight with no code change?
Yes, when the same request meets a different serving condition. One test run passes, a later run under another load pattern fails, and nothing in your repository moved.
A screenshot records one output and none of the state that produced it. A useful incident report captures the model identifier, request payload, decoding settings, output limit, timestamp, response metadata, and run number, plus tool calls and arguments for agent failures.
What happens to caching and A/B tests?
Caching assumes one cache key stands for an interchangeable result. A cache returns a stable stored completion while cache misses expose the underlying variation, so hit and miss paths are quietly measuring different systems.
A/B tests inherit the problem whenever traffic or load differs between variants: an output-sensitive metric can move because one variant met different batch conditions. Randomized assignment spreads traffic around, but it does not remove infrastructure-dependent variation. If the metric is task success, run repeated trials and keep the execution conditions visible.
What should you do about it?
Measure the variation before you change prompts or blame the model. Repeat identical requests, compare the first divergent token or tool call, record execution metadata, separate cache hits from misses, report pass and fail counts alongside the run count, and ask your provider whether the serving system guarantees batch-invariant inference.
For self-hosted inference, deterministic kernels are an option with a real price. Thinking Machines Lab measured Qwen3-8B over 1,000 sequences at 90 to 110 output tokens: default vLLM took 26 seconds, an unoptimized deterministic implementation 55 seconds, and an improved attention kernel 42 seconds. With deterministic kernels all 1,000 completions matched.
Those numbers belong to that model, workload and implementation, and are not a universal penalty. On a hosted API, ask for deterministic serving controls rather than assuming temperature 0 supplies them.
One caution on diagnosis: a different answer proves only that the whole system did not reproduce itself. Prompt serialization, hidden state, model updates, routing, retrieval results, tool responses and caching can all change the effective computation. Temperature 0 rules out sampling at decode time and nothing before it, which is why reproducible evaluation needs a run protocol rather than a parameter value.
Why does AI give different answers to the same question?
The full inference path can change even when the visible prompt is identical. Server-side batching, routing, hidden context, retrieval results, tool responses, model updates, and decoding settings all affect the result. Temperature 0 removes sampling from token selection, but it does not guarantee identical numerical computation.
Does temperature 0 make an LLM deterministic?
Temperature 0 makes decoding greedy, selecting the highest-probability computed token. It does not guarantee deterministic logits. Thinking Machines Lab found that changing batch size changed reduction order and produced different temperature-0 outputs for Qwen3-235B-Instruct.
What is LLM nondeterminism?
LLM nondeterminism is output variation across repeated requests with the same visible inputs and settings. The cause can be sampling, numerical differences, routing, hidden state, retrieval, tools, caching, model changes, or batch-dependent inference. The term describes observed behavior, not one specific mechanism.
How should I evaluate an LLM that gives different answers?
Run each case repeatedly, record every output, and report the run count with pass and fail counts. Compare the first different token, tool call, or branch rather than the final strings. Keep model identifiers, decoding settings, timestamps, cache conditions, and response metadata with the result.
How can I make LLM inference reproducible?
Use a serving implementation with deterministic or batch-invariant kernels, fix the model and inference configuration, control retrieval and tool inputs, separate cache hits from misses, and repeat your test cases. Deterministic kernels impose a workload-dependent cost, so measure it rather than assuming a universal value.