Autoregressive Decoding

Decoder-only generation produces one token at a time: previous tokens become context for the next token.

Decoder loop

1. Start with <BOS> or user prompt.
2. Run model and get next-token probabilities.
3. Select token (greedy/beam/sampling).
4. Append token and repeat.
5. Stop at <EOS> or max length.

Decoding strategies

Method	Behavior	Trade-off
Greedy	Pick best token each step	Fast, deterministic, can be myopic
Beam search	Track top candidate sequences	Better global quality, slower
Top-k sampling	Sample from top-k tokens	Controlled diversity
Top-p sampling	Sample from dynamic probability mass	Adaptive diversity

Common generation controls

• Temperature: lower for factual, higher for creative.
• Repetition/frequency penalty: reduce loops and repeated phrases.
• max_tokens + stop sequences: hard output boundaries.