Temperature is the single most important sampling parameter when working with large language models. It controls the randomness — or more precisely, the entropy — of the model’s token selection process. Every time an LLM generates the next token, it produces a probability distribution over its entire vocabulary (often 30,000–100,000+ tokens). Temperature determines how “peaked” or “flat” that distribution is before sampling.
Understanding temperature is essential for AI engineers because it directly affects output quality, consistency, and appropriateness for different use cases. Setting temperature too high produces incoherent, hallucination-prone text. Setting it too low produces repetitive, uncreative output. The right temperature depends entirely on the task — and knowing how to tune it is a fundamental skill.
The name “temperature” comes from statistical mechanics, where temperature controls the randomness of particle states in a thermodynamic system. In the Boltzmann distribution, higher temperature means particles are more uniformly distributed across energy states — analogous to how higher LLM temperature spreads probability more uniformly across tokens.
To understand temperature, you first need to understand the token generation pipeline. When an LLM processes a prompt, the final layer outputs a vector of raw scores called logits — one score for every token in the vocabulary. These logits are not probabilities; they can be any real number (positive, negative, or zero). To convert logits into a probability distribution, the model applies the softmax function.
P(token_i) = exp(logit_i / T) / Σ_j exp(logit_j / T)
Where T is the temperature parameter, logit_i is the raw score for token i, and the sum runs over all tokens in the vocabulary. When T = 1, this is the standard softmax function.
The key insight is that temperature divides the logits before softmax is applied. This scaling changes the shape of the resulting probability distribution:
T = 1.0 (Default): The standard softmax is applied with no modification. The probability distribution reflects the model’s learned preferences exactly as trained. This is the baseline behavior.
T < 1.0 (Low Temperature): Dividing logits by a number less than 1 amplifies the differences between them. If the top token had logit 5.0 and the second had 3.0, dividing by 0.5 gives 10.0 and 6.0 — the gap widens from 2 to 4. After softmax, the top token gets an even larger share of probability. The distribution becomes “sharper” and more deterministic.
T > 1.0 (High Temperature): Dividing by a number greater than 1 compresses the differences between logits. The same 5.0 and 3.0 divided by 2.0 become 2.5 and 1.5 — the gap shrinks from 2 to 1. After softmax, probability is distributed more evenly across tokens. The distribution becomes “flatter” and more random.
T → 0 (Greedy Decoding): As temperature approaches zero, the distribution collapses to a single spike on the highest-probability token. This is equivalent to always picking the argmax — called greedy decoding. Output is completely deterministic.
Consider a model generating the next word after “The cat sat on the”. Suppose the top 5 logits are:
Token | Logit | T=0.3 Prob | T=1.0 Prob | T=2.0 Prob
----------|-------|------------|------------|----------
"mat" | 5.2 | 89.4% | 38.5% | 24.8%
"floor" | 4.1 | 7.8% | 21.4% | 19.1%
"roof" | 3.5 | 1.8% | 14.7% | 16.2%
"table" | 3.0 | 0.6% | 10.6% | 14.0%
"bed" | 2.8 | 0.3% | 8.2% | 12.8%At T=0.3, “mat” dominates with 89.4% probability — the output is nearly deterministic. At T=1.0, there’s a reasonable spread. At T=2.0, the distribution is nearly uniform — any token could be selected, leading to more surprising (and potentially less coherent) output.
Temperature is typically applied first in the sampling pipeline, before other filters like top-k and top-p. This ordering matters: temperature reshapes the full distribution, then top-k/top-p truncate it. Using temperature alongside these other parameters gives you fine-grained control over the trade-off between creativity and coherence.
Most LLM APIs expose temperature as a simple parameter:
// OpenAI API example
const response = await openai.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: "Write a haiku about databases" }],
temperature: 0.9, // Creative task → higher temperature
});
// Anthropic API example
const response = await anthropic.messages.create({
model: "claude-sonnet-4-20250514",
max_tokens: 256,
messages: [{ role: "user", content: "Extract the date from this invoice" }],
temperature: 0.0, // Extraction task → deterministic
});Temperature ≠ creativity. A common misconception is that higher temperature makes the model “more creative.” In reality, it makes the model sample more uniformly from its vocabulary. This can produce novel word combinations, but it can also produce incoherent nonsense. True creativity often comes from better prompting at moderate temperatures, not from cranking temperature to the maximum.
Temperature 0 isn’t always deterministic across API calls. Due to floating-point non-determinism in GPU computation and potential batching effects, setting temperature to 0 may still produce slight variations across API calls. If you need bit-identical outputs, some providers offer a seed parameter alongside temperature 0.
When building production systems, always set temperature explicitly rather than relying on defaults. API defaults can change between model versions. Document your temperature choice with a comment explaining why that value was chosen for the specific use case.
P(token_i) = exp(logit_i / T) / Σ exp(logit_j / T). Temperature divides logits, amplifying or compressing differences between token probabilities.