Top-p sampling, also known as nucleus sampling, is a text generation strategy that dynamically selects the pool of candidate tokens based on their cumulative probability rather than a fixed count. Introduced in the 2019 paper “The Curious Case of Neural Text Degeneration” by Holtzman et al., it was designed to address a fundamental weakness in earlier decoding methods: greedy decoding produces repetitive text, while top-k sampling uses a rigid cutoff that ignores how confident the model actually is at each step.
The core insight behind nucleus sampling is that the shape of the probability distribution over next tokens changes dramatically from step to step. Sometimes the model is highly confident and a single token carries 90% of the probability mass. Other times, dozens of tokens are plausible. A fixed-size candidate pool (like top-k) cannot adapt to this variance — nucleus sampling can.
For AI engineers, understanding top-p is essential because it is one of the most commonly exposed generation parameters across all major LLM APIs, including OpenAI, Anthropic, Google, and open-source inference engines like vLLM and llama.cpp. Choosing the right value of p directly impacts output quality, creativity, and reliability.
At each generation step, the model produces a probability distribution over its entire vocabulary (often 32,000–128,000+ tokens). Nucleus sampling then applies the following procedure:
p.The nucleus is the smallest set of tokens whose cumulative probability mass exceeds the threshold p. Only tokens inside the nucleus are eligible for selection. Setting top_p=0.9 means: “Consider only the most probable tokens that together account for 90% of the likelihood.”
Consider a model predicting the next word after “The cat sat on the”. Suppose the probability distribution looks like this:
Token Probability Cumulative
"mat" 0.35 0.35
"floor" 0.20 0.55
"roof" 0.15 0.70
"bed" 0.10 0.80
"table" 0.07 0.87
"chair" 0.05 0.92 ← exceeds p=0.9
"window" 0.03 0.95
"fence" 0.02 0.97
...remaining tokens...With top_p=0.9, the nucleus includes {“mat”, “floor”, “roof”, “bed”, “table”, “chair”} — six tokens. Their probabilities are renormalized to sum to 1.0, and one is sampled. The remaining tokens (“window”, “fence”, etc.) are excluded entirely.
If the model were more confident — say “mat” had probability 0.92 — then the nucleus would contain just one token, and the output would be deterministic. This adaptive behavior is nucleus sampling’s key advantage.
import torch
import torch.nn.functional as F
def top_p_sample(logits, p=0.9, temperature=1.0):
"""Sample from a distribution using nucleus (top-p) sampling."""
# Apply temperature scaling
scaled_logits = logits / temperature
probs = F.softmax(scaled_logits, dim=-1)
# Sort probabilities in descending order
sorted_probs, sorted_indices = torch.sort(probs, descending=True)
# Compute cumulative probabilities
cumulative_probs = torch.cumsum(sorted_probs, dim=-1)
# Create mask: remove tokens with cumulative prob above threshold
# Shift right so the first token above threshold is kept
sorted_mask = cumulative_probs - sorted_probs >= p
sorted_probs[sorted_mask] = 0.0
# Renormalize
sorted_probs /= sorted_probs.sum()
# Sample from filtered distribution
sampled_index = torch.multinomial(sorted_probs, 1)
token_id = sorted_indices[sampled_index]
return token_idTemperature and top-p are often used together, but they operate at different stages. Temperature scales the raw logits before softmax, reshaping the overall distribution. Top-p then filters after softmax, pruning the tail. The interaction matters:
temperature=1.2, top_p=0.9): Temperature flattens the distribution, so more tokens enter the nucleus. Output is highly creative and diverse.temperature=0.3, top_p=0.95): Temperature concentrates probability on top tokens, so the nucleus is small despite a generous p. Output is focused and predictable.Both methods restrict the candidate pool, but they differ fundamentally in how:
k tokens regardless of the probability landscape. With k=50, if the top token has 99% probability, you still consider 49 near-zero alternatives. Conversely, if probability is evenly spread, 50 tokens might cover only 10% of the mass.For most production applications, start with top_p=0.9 to 0.95 and temperature=0.7. For structured/deterministic outputs (JSON, code), lower to top_p=0.1 to 0.5. For creative writing, raise to top_p=0.95 to 1.0.
Every major LLM API exposes top_p as a parameter. In the OpenAI API, it appears as top_p in the chat completions endpoint. Anthropic’s Claude API uses top_p in the messages endpoint. Google’s Gemini API offers it as topP. Understanding what this parameter does is essential for tuning generation quality in production.
top_p=0.3-0.5 — Keep responses focused, accurate, and on-brand. Avoid hallucinated or off-topic responses.top_p=0.9-1.0 — Allow diverse vocabulary and unexpected word choices for marketing copy, storytelling, or brainstorming.top_p=0.1-0.3 — Prioritize syntactically correct and semantically precise token sequences. Low values prevent the model from choosing unusual variable names or syntax.top_p=0.5-0.7 — Balance between faithfulness to source material and natural readability.In real systems, top-p is often used alongside other controls:
Research from the original nucleus sampling paper showed that with greedy decoding, neural text generation degenerates into repetitive loops within a few hundred tokens. Top-k with k=40 reduced but did not eliminate this. Top-p with p=0.95 produced text with a perplexity and token distribution closest to human-written text, as measured by self-BLEU diversity scores and human evaluation.
p, then samples from that set.0.1 (deterministic) to 1.0 (full distribution), with 0.9 being a common default.top_p as a configurable parameter — understanding it is essential for production AI engineering.