Tokens, Context Window, Temperature, Top-p

GenAI bills and bugs both start with the same unit: the token. Once you see prompts, context limits, and sampling knobs in token terms, pricing, truncation failures, and “why did it get weird?” stop feeling mystical.

Intuition

The model never reads “words.” A tokenizer chops your string into discrete IDs from a fixed vocabulary. Those IDs fill a context window — a hard budget shared by system instructions, history, retrieved docs, tools, and the reply you hope to get.

After the model scores the next token, temperature and top-p reshape which candidates you are willing to sample. Low and narrow → boring but stable. High and wide → spicy but risky. Production systems treat these as reliability controls, not creativity cosmetics.

How it works

Tokens

Tokenization is model-specific. Rough patterns for English:

• Common words and punctuation often map to one token.
• Rare words, code identifiers, and non-English scripts may split into many pieces.
• Whitespace and formatting count.

Rule-of-thumb for planning (not billing truth): ~4 characters per token for English prose, or ~100 tokens ≈ 75 words. Always measure with the real tokenizer in CI for anything cost-sensitive.

Context window

If the window is W tokens, everything the model can attend to in one call must fit in W (API products sometimes split “input” vs “output” limits — read the docs for your provider). Overflow typically means:

• Silent truncation of the oldest or middle content
• API errors
• Worse: your safety policy or schema falls out of view

Long RAG dumps are the classic way to spend the whole budget on noise. Prefer ranked snippets, summaries, and hard caps per source.

Temperature

Let z_i be the logit for vocabulary item i. Softmax turns logits into probabilities. Temperature T > 0 scales logits before softmax:

p_i = softmax(z_i / T)

• T → 0 (practically very small): distribution collapses toward the argmax → greedy-like, stable.
• T = 1: use the model’s native distribution.
• T > 1: flatten the distribution → more surprise, more nonsense risk.

Temperature does not add knowledge. It only changes how aggressively you explore the model’s uncertainty.

Top-p (nucleus sampling)

Sort tokens by probability descending. Keep the smallest prefix whose cumulative probability is at least p. Sample only inside that nucleus.

• top_p = 0.1 → tiny, high-confidence set
• top_p = 0.9 → broader, still cuts the long tail
• top_p = 1.0 → effectively no nucleus cutoff

Top-p adapts to the shape of the distribution: when the model is peaked, the nucleus is small; when it is flat, the nucleus grows.

Practical presets

Tune one primary diversity control first (often temperature or top-p), then the other. Extreme values on both compound into chaos.

Cost and latency mental model

Every call roughly costs price_in * prompt_tokens + price_out * completion_tokens. Output tokens are often more expensive and always more latency-sensitive because generation is sequential. That is why short, structured answers beat essay-length chat for automation: you pay less and finish sooner. Caching identical system prompts (when your provider supports it) and trimming history aggressively are usually bigger wins than shaving a few words from a user template.

Interaction with tools and RAG

Retrieved chunks and tool schemas sit in the same window as the question. A 20-tool catalog with verbose descriptions can crowd out the user turn. Prefer concise tool descriptions, retrieve top-k passages not whole manuals, and summarize old turns instead of replaying full transcripts forever.

In code

Approximate English token count without a real tokenizer — good for budgeting demos, bad for invoices:

def estimate_tokens(text: str) -> int:
    # Rough: ~4 chars/token for English prose
    return max(1, (len(text) + 3) // 4)


prompt = "System: be brief.\nUser: Explain tokens."
print(estimate_tokens(prompt))

Simulate a context packer that reserves output room:

def pack(parts: list[str], window: int, reserve_out: int) -> list[str]:
    budget = window - reserve_out
    kept = []
    used = 0
    for part in reversed(parts):  # keep newest first
        cost = estimate_tokens(part)
        if used + cost > budget:
            continue
        kept.append(part)
        used += cost
    return list(reversed(kept))


history = ["old policy...", "mid turn...", "latest user question"]
print(pack(history, window=200, reserve_out=50))

Temperature on a toy logit vector (numpy-free):

import math


def softmax(xs):
    m = max(xs)
    exps = [math.exp(x - m) for x in xs]
    s = sum(exps)
    return [e / s for e in exps]


def with_temperature(logits, T: float):
    return softmax([z / T for z in logits])


logits = [2.0, 1.0, 0.1]  # pretend vocab of 3
print([round(p, 3) for p in with_temperature(logits, 0.5)])  # peakier
print([round(p, 3) for p in with_temperature(logits, 1.5)])  # flatter

Nucleus filter:

def nucleus(probs: list[float], p: float) -> list[int]:
    order = sorted(range(len(probs)), key=lambda i: probs[i], reverse=True)
    kept, cum = [], 0.0
    for i in order:
        kept.append(i)
        cum += probs[i]
        if cum >= p:
            break
    return kept


probs = [0.5, 0.3, 0.15, 0.05]
print(nucleus(probs, 0.8))  # indices covering >= 0.8 mass

What goes wrong

• Counting words as tokens — code and CJK blow past naive estimates; you hit limits or surprise bills.
• Filling context with raw PDFs — the answer evidence is present in the file but not in the top of the packed window.
• High temperature for factual JSON — field names mutate; parsers fail.
• Stacking high T and high top-p — you sample the junk tail.
• Forgetting output reservation — a 128k window with 120k of prompt leaves a tiny completion budget.
• Assuming temperature 0 is fully deterministic — floating-point and batching can still yield rare ties or infra differences; treat “low T” as more stable, not formally deterministic across hosts.

One-line summary

Tokens meter cost and memory; the context window is a shared budget; temperature and top-p reshape sampling — use them deliberately for stability vs diversity.

Key terms

• Token — Vocabulary piece the model reads/writes; billing and limits are usually token-based.
• Tokenizer — Algorithm mapping text ↔ token IDs for a model family.
• Context window — Max tokens attendable in one model call.
• Temperature — Softmax temperature that sharpens or flattens next-token probabilities.
• Top-p / nucleus sampling — Sample from the smallest set of tokens whose cumulative probability ≥ p.
• Logit — Raw score for a vocabulary item before softmax.
• Truncation — Dropping content that does not fit the context budget.

)