Prompting and model APIs

Calling a model well: the request shape, system prompts, clear instructions and examples, streaming, handling failures and treating prompts as code.

Chat messages API

Also called messages.create, chat completions, LLM API call

What: The core request: a model id, a maximum output length, an optional system prompt and a list of user and assistant messages. The response has content blocks, a stop reason and token usage. The API is stateless, so you resend history every turn.
Why: Every feature from a chatbot to an agent is built from this one request shape, so knowing its fields by heart makes every SDK and framework easier to read.

Python

from anthropic import Anthropic

client = Anthropic()
msg = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    system="You are a concise support assistant.",
    messages=[{"role": "user", "content": question}],
)
print(msg.stop_reason, msg.usage.output_tokens)
print("".join(b.text for b in msg.content if b.type == "text"))

Watch out

Reading only the text and ignoring stop_reason: a value of max_tokens means the answer was cut off.

System prompt

Also called system message, instructions, developer message

What: Instructions sent separately from the conversation that set the role, rules, tone, output format and background for every turn.
Why: It is where stable product behaviour lives. Keeping it identical across requests and at the start of the prompt also makes it cacheable.

Watch out

Pasting untrusted user or document text into the system prompt, which gives it the authority of your own instructions.

Clear, specific instructions

Also called prompt engineering, output format

What: Stating the task, audience, constraints, what good output looks like and the exact format, as you would brief a capable colleague with no context. Wrap inputs in clear delimiters such as <document> tags.
Why: Most bad outputs come from underspecified prompts, not weak models. Explaining why a rule exists helps the model apply it sensibly to cases you did not foresee.
Example: Instead of "Summarise this", write "Summarise this support ticket in 3 bullet points for an on-call engineer: symptom, affected customers and error codes. Output only the bullets."

Watch out

Shouting rules in capitals with ALWAYS and NEVER: modern models over-apply emphatic instructions.

Few-shot examples

Also called in-context learning, one-shot, examples in prompts

What: Including a handful of worked input and output examples in the prompt so the model copies their format, style and judgement.
Why: Examples pin down format and edge-case handling faster than long descriptions, especially for classification and extraction.

Watch out

Using near-identical examples: the model copies their surface details, so vary them and include the tricky cases.

Streaming responses

Also called SSE, server-sent events, token streaming

What: Receiving output piece by piece as it is generated, over server-sent events, instead of waiting for the whole response.
Why: Users see text within a second instead of a spinner, and long generations avoid HTTP timeouts.

Python

with client.messages.stream(
    model="claude-sonnet-5",
    max_tokens=4096,
    messages=[{"role": "user", "content": prompt}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
    final = stream.get_final_message()

Watch out

Feeding streamed chunks of JSON straight to a parser: partial output is invalid until the end, so parse the final message.

Timeouts, retries and backoff

Also called exponential backoff, retry, timeout

What: Setting a timeout on every call and retrying transient failures (429, 5xx, overloaded, dropped connections) with exponential backoff and jitter. Official SDKs retry a couple of times by default.
Why: Model APIs are slower and fail more often than typical APIs, and without limits one hung call ties up a worker or a waiting user.

Python

import anthropic

client = anthropic.Anthropic(max_retries=3, timeout=30.0)
try:
    msg = client.messages.create(**request)
except anthropic.RateLimitError:
    queue_for_later(request)
except anthropic.APIStatusError as err:
    logger.error("model call failed: %s", err.status_code)
    raise

Watch out

Retrying 400 errors, which fail the same way every time, or retrying without a cap so an outage multiplies your traffic.

Hosted vs local models

Also called self-hosted models, Ollama, local LLM, local model

What: Hosted models run behind a provider API. Open-weight models such as Llama, Qwen, Gemma or Mistral can run on your own hardware with tools like Ollama, llama.cpp or vLLM.
Why: Hosted gives the strongest models with no infrastructure. Local suits strict data residency, offline use, free experimentation or very high volumes of simple tasks.

Watch out

Comparing only token prices: GPUs, operations and a lower quality ceiling often make self-hosting the more expensive option.

Prompts in version control

Also called prompt management, prompt registry

What: Treating prompts as code: stored in the repository or a prompt registry, reviewed, versioned and linked to the eval results each version produced.
Why: A one-word change can shift behaviour widely, so you need to know which prompt produced which output and be able to roll back.

Watch out

Editing prompts live in a dashboard with no history and no eval run.

Open-weight models

Also called open weights, open models, quantised models

What: Models whose trained weights are published for download, each under its own licence. You can run them on a laptop (Ollama), on your own GPU server (vLLM) or through a hosted provider that serves them behind an API.
Why: They give control over data, cost at volume and fine-tuning. Choosing between models is covered at sefism.com/roadmaps/choosing-llms/.

Watch out

Assuming open weights means open source: some licences restrict commercial use, so read the licence before shipping.

Go deeper