Tokens, Context Window, Temperature, Top-p

Tokens

Models process text as tokens, not full words. A token can be a word, part of a word, punctuation, or symbol.

Why it matters: usage limits and pricing are often token-based.

Context window

The context window is the maximum tokens the model can consider in one request (prompt + conversation + retrieved docs + output).

• If the window is full, old content is dropped.
• Long prompts can reduce space for model output.

Temperature

Controls randomness in token selection.

• Low temperature: consistent and focused outputs.
• High temperature: more diverse, creative outputs.

Top-p (nucleus sampling)

Model selects next token from the smallest set whose cumulative probability is at least p.

• Lower top-p narrows choices.
• Higher top-p allows broader choices.

Practical settings

Task	Temperature	Top-p
Policy/compliance answers	Low	Low to medium
Brainstorming ideas	Medium to high	Medium to high
Code generation	Low to medium	Medium