Task Prompting: Summarize, QA, Classify

Most business LLM traffic is not open-ended creativity. It is summarize this, answer from that, or label as one of these. Task prompting means picking the right shape of instructions, context, and output contract for each job so models stay useful and your validators stay boring.

Intuition

Think in verbs, not vibes.

Each verb wants a different system prompt, a different temperature (usually low), and a different success test. Reusing one “helpful assistant” blob for all three is how summaries grow citations that were never in the source and classifiers emit poetry.

How it works

Summarization

Goal. Shorter artifact that preserves decisions, numbers, and owners the reader needs.

Prompt pattern.

  1. Role: “summarizer for busy operators.”
  2. Constraints: length budget (sentences or tokens), audience, must-keep fields (dates, amounts, action items).
  3. Ban: new facts not in the source; hedging filler.
  4. Structure: bullets or sections the UI already knows how to render.

Eval signals. Compression ratio band; presence of required entities from a checklist; hallucination spot-check (claim → span in source). Prefer extractive anchors for numbers (“$4,200 as written in source”).

Variants. Executive brief vs changelog vs “action items only.” Do not ask one prompt to be all three without an explicit mode flag.

Question answering

Goal. Correct answer grounded in provided context (closed-book only when you truly want parametric knowledge).

Prompt pattern.

  1. Separate CONTEXT from QUESTION with clear delimiters.
  2. Instruct abstention: if context is insufficient, say so and list what is missing.
  3. Ask for short answers plus optional citation markers ([doc_id]) when RAG-fed.
  4. Temperature near 0 for factual QA.

Eval signals. Exact match / token F1 against gold for short answers; faithfulness rubrics for long ones; refusal correctness on unanswerable items. Include adversarial context that almost-but-not-quite answers the question.

Classification

Goal. Map input → label from a fixed taxonomy (plus optional confidence / secondary tags).

Prompt pattern.

  1. List labels with one-line definitions and boundary examples.
  2. Demand machine-readable output: JSON {"label": "...", "confidence": 0.0} or a single enum token.
  3. Provide 3–8 few-shot edge cases that used to confuse humans.
  4. Include an other / needs_review bucket so the model is not forced to lie.

Eval signals. Accuracy / macro-F1 on a labeled set; calibration of confidence; rate of invalid labels (should be ~0 with validation + repair).

flowchart TB T{Task verb} T -->|summarize| S[Length + must-keep + no new facts] T -->|qa| Q[Context bounds + abstain + cite] T -->|classify| C[Closed labels + schema + other] S --> V[Validate / grade] Q --> V C --> V

Shared discipline

In code

Three tiny prompt builders and a classifier validator — the skeleton you extend with real model calls.

from dataclasses import dataclass
import json
import re

LABELS = {"billing", "shipping", "product", "other"}

def prompt_summarize(source: str, max_bullets: int = 5) -> list[dict]:
    return [
        {"role": "system", "content": (
            "Summarize for an on-call engineer. Use at most "
            f"{max_bullets} bullets. Keep every dollar amount and deadline "
            "exactly as written. Do not add facts absent from the source."
        )},
        {"role": "user", "content": f"SOURCE:\n{source}\n\nWrite the summary."},
    ]

def prompt_qa(context: str, question: str) -> list[dict]:
    return [
        {"role": "system", "content": (
            "Answer only using CONTEXT. If CONTEXT is insufficient, reply "
            'exactly: INSUFFICIENT_CONTEXT. Otherwise answer in <= 3 sentences '
            "and cite chunk ids like [c1] when used."
        )},
        {"role": "user", "content": f"CONTEXT:\n{context}\n\nQUESTION:\n{question}"},
    ]

def prompt_classify(text: str) -> list[dict]:
    defs = (
        "billing: invoices, refunds, charges\n"
        "shipping: delivery, tracking, address\n"
        "product: features, bugs, how-to\n"
        "other: none of the above or unclear"
    )
    return [
        {"role": "system", "content": (
            "Classify the ticket. Labels:\n"
            f"{defs}\n"
            'Return JSON only: {"label": "<one label>", "confidence": 0-1}'
        )},
        {"role": "user", "content": text},
    ]

@dataclass
class ClassResult:
    label: str
    confidence: float

def parse_classify(raw: str) -> ClassResult:
    # strip optional fences
    cleaned = re.sub(r"^```(?:json)?|```$", "", raw.strip(), flags=re.I | re.M).strip()
    obj = json.loads(cleaned)
    label = str(obj["label"]).lower().strip()
    conf = float(obj["confidence"])
    if label not in LABELS:
        raise ValueError(f"invalid label {label}")
    if not 0.0 <= conf <= 1.0:
        raise ValueError("confidence out of range")
    return ClassResult(label, conf)

# Demo without a model: validate a well-formed classifier output
print(parse_classify('{"label": "billing", "confidence": 0.88}'))

Wire prompt_* into your serving client; on parse_classify failure, retry once with the validator error text, then fall back to other / human review.

What goes wrong

Prefer bullets for summarize, shortest correct span for QA, and label-only (rationale offline) for classify. Combined tasks (“summarize then urgency”) should be two validated calls until each field is stable alone.

One-line summary

Task prompting means matching summarize / QA / classify to explicit contracts—compression with no new facts, grounded answers with abstention, and closed labels with validation—not one generic chat prompt.

Key terms