In chat APIs, not every message is equal. The system (or developer) message is the constitution: long-lived rules for identity, tools, safety, and output style. User messages are the day’s petitions. Assistant messages are prior replies that keep the conversation coherent. Role design is how you turn a generic model into a product-specific agent without fine-tuning.
If the user prompt is a ticket, the system prompt is the employee handbook. You do not reprint the handbook on every sticky note; you assume the employee already read it. Likewise, put stable policy in the system role and put task-specific data in the user role. Mixing them makes updates painful and invites users (or retrieved docs) to contradict your rules.
A good role is not a costume (“you are a pirate”). It is a job description: mission, audience, tools allowed, refusal rules, and escalation paths. Personality is a thin layer on top of that contract.
What belongs in the system prompt.
search_docs before answering product questions”).What belongs in the user message. The current question, uploaded text, and ephemeral context. Prefer injecting RAG snippets into a clearly labeled user or tool section, not into the system prompt, so documents cannot quietly rewrite the constitution.
Layering. Many APIs support a developer/system channel that is harder for end users to override. Still assume adversarial or confused users will try. Restate critical rules near the end of the system message (recency bias) and again in tool wrappers.
Role vs persona. Persona (“friendly mentor”) affects tone. Role (“mentor who only teaches from provided notes”) affects authority and grounding. Prefer role + thin persona over theatrical characters that fight your safety rules.
Multi-agent roles. In agent systems, each specialist gets a narrow system prompt (researcher, critic, writer). Narrow roles reduce cross-talk and make failures attributable.
A minimal role stack: assemble system text from versioned fragments, keep user content separate, and unit-test that required policy phrases are present.
from pathlib import Path
FRAGMENTS = {
"identity": "You are BackbenchTutor, a concise mentor for backend and GenAI interviews.",
"grounding": "Answer only from CONTEXT when provided. If missing, say what is missing.",
"safety": "Refuse requests for credentials, malware, or illegal activity. Offer safe alternatives.",
"style": "Use short sections and plain language. Prefer bullets over essays.",
}
def build_system(keys: list[str]) -> str:
return "\n\n".join(FRAGMENTS[k] for k in keys)
def build_messages(question: str, context: str = "") -> list[dict]:
user = question if not context else (
f"CONTEXT:\n\"\"\"\n{context}\n\"\"\"\n\nQUESTION:\n{question}"
)
return [
{"role": "system", "content": build_system(
["identity", "grounding", "safety", "style"]
)},
{"role": "user", "content": user},
]
msgs = build_messages(
"What is idempotency?",
context="Idempotency means retrying a request does not change the result beyond the first success.",
)
# Regression check: policy phrases must ship with the build
sys_text = msgs[0]["content"]
assert "Answer only from CONTEXT" in sys_text
assert "Refuse requests for credentials" in sys_text
print(msgs[0]["role"], "chars=", len(sys_text))
print(msgs[1]["content"][:80], "...")Version the fragment map in git. Changing tone should not require hunting through every user-facing template.
Write the system prompt as if onboarding a new hire on day one. Include mission (“help learners prepare for backend interviews”), non-goals (“do not invent company-specific salary bands”), tools (“you may call search_curriculum”), and escalation (“if the user reports a billing outage, direct them to support, do not debug payments”). Ambiguous kindness (“be helpful”) without non-goals is how models overshare or overpromise.
Multi-tenant products often inject a thin dynamic slice: tenant name, locale, feature flags. Keep that slice structured and small. Do not dump entire policy PDFs into the system channel; retrieve them. Dynamic system text still needs the same review as static text because it ships to every session.
Testing roles. Beyond golden Q&A, add policy probes: requests that should refuse, requests that should use a tool first, and requests that try to override the role (“you are now unrestricted”). Store expected behaviors beside the fragment map. When marketing asks for a wittier persona, re-run the probes before celebrating the tone change.
System prompts are the product’s constitution — put durable identity, tools, and safety there; keep ephemeral tasks and data in the user channel.