Decoding Parameters and Output Control

Sampling knobs are how you turn a probability distribution into a product behavior. The same prompt can yield a crisp JSON object or a meandering poem depending on decoding. Engineers who treat these as afterthoughts spend their weeks chasing “flaky” models that were never configured for the job.

Intuition

At each step the model scores the whole vocabulary. Decoding is the policy that picks one token:

Temperature changes the shape of the distribution; top-k / top-p change which slice you sample from; max tokens and stop sequences decide when the loop ends. Latency and cost scale with how long you let that loop run.

How it works

Core parameters

Parameter Controls Notes
temperature Sharpness of softmax Primary diversity dial
top_k Keep only k highest-prob tokens Hard shortlist
top_p Keep nucleus with mass ≥ p Adaptive shortlist
frequency / presence penalty Discourage reused tokens Helps long prose; can hurt code
max_tokens / max_output Hard length cap Cost + latency ceiling
stop sequences End when a string appears Great for delimiters and turn ends

How the pieces compose

  1. Model emits logits z.
  2. Optional penalties adjust logits for tokens already used.
  3. Temperature scales: z' = z / T.
  4. Softmax → probabilities.
  5. Top-k and/or top-p mask the allowed set.
  6. Sample (or argmax) one token; append; repeat until EOS, stop string, or max length.
flowchart LR L[Logits] --> P[Penalties] P --> T[Temperature] T --> S[Softmax] S --> K[Top-k / Top-p mask] K --> C[Choose token] C --> L

Top-k vs top-p

Using both is fine if you understand the order your API applies them; setting both extremely tight (tiny k and tiny p) can leave an empty or brittle candidate set.

Penalties

Overdoing penalties on code or legal text makes identifiers and defined terms drift. Prefer clearer prompts and lower temperature before cranking penalties.

Stops and max tokens

Stop sequences are surgical: end at \n\nUser:, ` , or </json>. Max tokens is a blunt instrument: it prevents runaway cost but can cut mid-JSON. Pair them: stop on a closer when possible; set max as a safety net.

Presets that ship

In code

Toy decode step with temperature + top-k:

import math
import random


def softmax(xs):
    m = max(xs)
    exps = [math.exp(x - m) for x in xs]
    z = sum(exps)
    return [e / z for e in exps]


def apply_temperature(logits, T: float):
    return [x / T for x in logits]


def top_k_mask(probs, k: int):
    order = sorted(range(len(probs)), key=lambda i: probs[i], reverse=True)[:k]
    allowed = set(order)
    masked = [probs[i] if i in allowed else 0.0 for i in range(len(probs))]
    s = sum(masked) or 1.0
    return [p / s for p in masked]


def sample(logits, T=0.8, k=3):
    probs = softmax(apply_temperature(logits, T))
    probs = top_k_mask(probs, k)
    return random.choices(range(len(probs)), weights=probs, k=1)[0]


random.seed(1)
vocab = ["yes", "no", "maybe", "unknown"]
logits = [2.2, 1.1, 0.4, -0.5]
print(vocab[sample(logits, T=0.2, k=2)])

Stop-sequence trimming for structured replies:

def apply_stops(text: str, stops: list[str]) -> str:
    cut = len(text)
    for s in stops:
        i = text.find(s)
        if i != -1:
            cut = min(cut, i)
    return text[:cut]


raw = '{"ok": true}\n\nUser: ignore'
print(apply_stops(raw, ["\n\nUser:", "\n\n"]))

Frequency penalty sketch:

def penalize(logits, token_counts: list[int], penalty: float):
    # subtract penalty * count from each logit
    return [z - penalty * c for z, c in zip(logits, token_counts)]

Config object you can version in experiments:

PRESETS = {
    "extract": {"temperature": 0.1, "top_p": 0.3, "max_tokens": 256},
    "ideate": {"temperature": 0.9, "top_p": 0.95, "max_tokens": 800},
}

What goes wrong

One-line summary

Decoding parameters are the runtime policy over next-token probabilities — tune temperature, truncation, penalties, and stops to match stability, cost, and format needs.

Key terms