What this is: An LLM does not emit a paragraph in one shot. It runs an autoregressive loop: read the prompt, score the next token, append the choice, repeat until a stop condition.
Why it matters: Architecture (causal Transformer) makes that loop valid; decoding strategy decides whether you get a safe factual tone, a creative draft, or a repetitive spiral. If you ship GenAI features, this lesson is the difference between "the model is dumb" and "we sampled badly."
Think of improv storytelling: each sentence must fit everything already said. The model outputs a logit vector over the vocabulary — raw, unnormalized scores. Softmax turns logits into probabilities. A policy picks a token ID. That ID becomes part of the context for the next round.
There is no separate "planning module" unless you add tools or search on top — the default product path is this token treadmill.
| Strategy | Plain-English idea | Trade-off |
|---|---|---|
| Greedy | Always pick the highest-probability token | Fast and stable, but often bland or locally trapped |
| Beam search | Keep the top-B partial strings alive | Better global score, less diversity, heavier compute |
| Top-k | Sample only from the k largest probabilities | Fixed-size candidate set |
| Top-p (nucleus) | Keep the smallest set whose probabilities sum to at least p | Adaptive support; common in chat APIs |
T (logits / T): T < 1 sharpens; T > 1 flattens.max_tokens.P(token_t | token_1..t-1) = softmax(logits_t / T)The full sequence probability factorizes as:
P(x_1..x_L) = product_t P(x_t | x_1..x_{t-1})Each token depends only on everything before it — matching the causal mask from the previous lesson.
| Method | Rule | Best for |
|---|---|---|
| Greedy | argmax P |
Fast, deterministic, factual tasks |
| Beam search | Keep top-B partial strings | Short, exact tasks (some translation) |
| Top-k | Sample from k largest probs | Controlled creativity |
| Top-p (nucleus) | Smallest set with mass ≥ p | Chat and open-ended generation |
Naively recomputing attention over the full prefix every step is wasteful. Inference stacks cache prior keys and values so each new token only pays for the new row — critical for product latency, even though the algorithm is still left-to-right.
Frequency and presence penalties (API-side) down-weight tokens already used, fighting loops like "Yes. Yes. Yes." They sit on the same logits the model just produced — product quality is model plus decoding policy.
A toy vocabulary and one decoding step with temperature, greedy, and top-p:
import numpy as np
def softmax(logits):
x = logits - logits.max()
e = np.exp(x)
return e / e.sum()
def top_p_filter(probs, p=0.9):
order = np.argsort(-probs)
sorted_p = probs[order]
cum = np.cumsum(sorted_p)
keep = cum <= p
keep[0] = True # always keep the top token
mask = np.zeros_like(probs, dtype=bool)
mask[order[keep]] = True
trimmed = np.where(mask, probs, 0.0)
return trimmed / trimmed.sum()
vocab = ["the", "cat", "sat", "mat", "because", "it", "<EOS>"]
logits = np.array([2.0, 3.5, 1.2, 0.4, 0.8, 1.5, -1.0])
for T in (0.5, 1.0, 1.5):
probs = softmax(logits / T)
print(f"T={T}:", dict(zip(vocab, np.round(probs, 3))))
probs = softmax(logits / 1.0)
print("greedy ->", vocab[int(np.argmax(probs))])
rng = np.random.default_rng(4)
nucleus = top_p_filter(probs, p=0.85)
token = rng.choice(len(vocab), p=nucleus)
print("top-p sample ->", vocab[token], "mass kept", nucleus[nucleus > 0].sum())Lower T concentrates mass on "cat"; higher T lifts the long tail. Nucleus sampling renormalizes after dropping the tail — the same knob exposed as top_p in many APIs.
max_tokens and stop sequences.Autoregressive decoding builds text token by token from causal next-token probabilities, with greedy, beam, top-k, and top-p policies shaping the trade-off between reliability and diversity.