Temperature and sampling

At each step, a language model produces a probability for every token in its vocabulary. Something then has to pick one. That's sampling, and a few parameters control it.

"The capital of France is" →  " Paris" 0.92   " a" 0.03   " the" 0.02   " located" 0.01 …

#The knobs

• Temperature rescales the probabilities before sampling.
  • Near 0: almost always pick the top token. The output is focused and repetitive, close to deterministic.
  • Around 1: sample from the model's actual distribution.
  • Above 1: flatten the distribution, so unlikely tokens get picked more often. More varied, and more likely to drift into nonsense.
• Top-p (nucleus sampling) samples only from the smallest set of tokens whose probabilities add up to p, such as 0.9, and cuts off the long tail of unlikely tokens.
• Top-k samples only from the k most likely tokens.

Adjust temperature or top-p, not both. Some newer models fix these values or ignore them, especially reasoning models, so check what your model supports.

#Choosing a setting

#Temperature 0 still isn't deterministic

Even at temperature 0, the same prompt can produce different outputs. Floating-point arithmetic on GPUs, batching with other requests, and model updates all introduce small differences, and once one token differs, everything after it can differ too. Don't rely on identical outputs:

• Validate the output instead of snapshot-testing exact text. See structured outputs.
• Run evals several times, or over enough cases that run-to-run noise averages out. See LLM-as-judge.
• Cache responses when you really need the same answer twice.

Temperature can't fix a vague prompt. If outputs are wrong rather than just varied, fix the instructions or add examples.