Prompts, roles and sampling

How you structure a request: who says what, which instructions persist, how history is kept, and which knobs change the style and length of the answer.

Messages and roles

Also called user, assistant, model

What: A conversation is a list of turns, each with a role: the user, and the model (called assistant, or model in Gemini). Earlier model turns are sent back so the model sees the history.
Why: The API is stateless per request unless you opt into stored state, so you control exactly what the model remembers.

Python

history = [
    {"role": "user", "content": "My name is Ada."},
    {"role": "assistant", "content": "Nice to meet you, Ada."},
    {"role": "user", "content": "What is my name?"},
]
response = client.responses.create(model=MODEL, input=history)

Watch out

Putting user-supplied text into the system or developer instructions, which gives it more authority than it deserves.

System prompts

Also called instructions, system instruction, developer message

What: Standing instructions that apply to the whole conversation: role, tone, rules, output format and context about the product.
Why: They keep behaviour consistent across turns and are the natural home for long, stable context that can be cached.

Python

response = client.responses.create(
    model=MODEL,
    instructions="You are Acme's support assistant. "
                 "Answer in British English, under 120 words.",
    input=question,
)

Watch out

Putting a timestamp or user id at the start of the system prompt, which changes every request and defeats prompt caching.

Keeping conversation state

What: Either resend the full history yourself each turn, or use a provider feature that keeps it for you where one exists.
Why: Managing history yourself gives full control (trimming, summarising, storing); provider-side state saves bandwidth and code.

Python

first = client.responses.create(
    model=MODEL, input="My name is Ada.")
second = client.responses.create(
    model=MODEL,
    previous_response_id=first.id,
    input="What is my name?",
)

Watch out

Letting history grow without limit; long chats get slower and costlier every turn.

Clear, structured instructions

What: State the task, the audience, the constraints and the output format explicitly, and separate instructions from data with headings or tags such as <document>.
Why: Models follow specific instructions far better than implied ones, and clear delimiters stop data being mistaken for instructions.
Example: Instead of "Summarise this", write "Summarise the ticket below for an on-call engineer in three bullet points: impact, cause, next step" and wrap the ticket in tags.

Watch out

Writing a vague prompt, then patching failures with ever more ALL CAPS rules.

Few-shot examples

Also called in-context examples

What: Including a handful of example inputs and ideal outputs in the prompt so the model copies the pattern.
Why: Examples pin down format and tone faster than paragraphs of description, especially for classification and extraction.

Watch out

Using examples that are all alike, so the model overfits to their length or wording. Vary them.

Temperature and sampling

Also called top_p, top_k

What: Temperature controls randomness when picking the next token: low values give focused, repeatable answers, higher values more varied ones. top_p and top_k limit which tokens can be picked.
Why: Use low values for extraction and classification, higher for brainstorming and creative text.

Python

response = client.responses.create(
    model=MODEL,
    input="Suggest three names for a coffee shop.",
    temperature=1.0,
    max_output_tokens=200,
)

Watch out

Several newer reasoning models reject or ignore sampling parameters (on Anthropic, Opus 5 and Sonnet 5 do); check the model docs and steer with instructions or effort.

Reasoning and effort

Also called thinking, reasoning models

What: Reasoning models think before answering, using extra hidden output tokens. You tune how much with a setting: OpenAI calls it reasoning effort, Anthropic effort with adaptive thinking, Gemini a thinking configuration.
Why: More thinking helps maths, code and multi-step tasks; less keeps simple routes fast and cheap.

Watch out

Using maximum effort everywhere. Measure on real traffic and lower it for routine requests.

Output limits and stop reasons

Also called max_tokens, finish_reason

What: You cap output length, and each response says why it stopped: finished naturally, hit the limit, called a tool or was declined.
Why: Checking the stop reason catches truncated JSON and cut-off answers before they reach users.

Python

response = client.responses.create(
    model=MODEL, input=prompt, max_output_tokens=300)
if response.status == "incomplete":
    print("Cut short:", response.incomplete_details.reason)

Watch out

On reasoning models, thinking counts against the output limit, so a tight cap can leave no room for the answer.

Go deeper