How LLM APIs work

An LLM API takes text (and sometimes images) in and returns generated text, billed by the token. These ideas shape every request you will write.

Tokens

What: Models read and write tokens: chunks of text that are often a word or part of one. In English a token averages roughly four characters, but code, numbers and other languages vary.
Why: Price, speed, context limits and output limits are all counted in tokens, so they are the unit you budget in.

Watch out

Estimating cost from word counts, or assuming two providers tokenise the same text identically.

Context window

What: The maximum number of tokens a model can take in one request, counting the system prompt, the conversation history, tool definitions, documents and the output.
Why: It limits how much history and reference material you can send; long contexts also cost more and can dilute the model’s attention.

Watch out

Resending an ever-growing chat history until requests fail or quietly get slow and expensive. Trim or summarise old turns.

generate_content and config

Also called client.models.generate_content, google-genai

What: The google-genai SDK exposes client.models.generate_content(model, contents, config). Options such as system instructions, temperature, tools and output schema go in config, typed with from google.genai import types.
Why: The same client works with the Gemini Developer API and with Vertex AI, so code moves between them with a change of client settings.

Watch out

Following examples for the older google-generativeai package, which has a different API.

Your first request

What: Create a client (it reads the API key from an environment variable) and send a prompt. The model name comes from configuration rather than the code.
Why: Keeping keys and model names out of source code lets you rotate keys and upgrade models without a code change.

Python

import os
from google import genai
from google.genai import types

client = genai.Client()  # reads GEMINI_API_KEY
MODEL = os.environ["MODEL"]

response = client.models.generate_content(
    model=MODEL,
    contents="Explain HTTP caching in two sentences.",
)
print(response.text)

Watch out

Hard-coding the API key in source or a notebook that later gets shared or committed.

Reading token usage

What: Every response reports how many input and output tokens it used (and cached tokens, where relevant).
Why: Logging usage per request and per feature is how you find what drives cost and catch runaway prompts early.

Python

u = response.usage_metadata
print(u.prompt_token_count, u.candidates_token_count)
print(u.total_token_count)
count = client.models.count_tokens(model=MODEL, contents=prompt)
print(count.total_tokens)

Choosing a model

What: Providers offer tiers: large models for hard reasoning and coding, mid-size for most product work, small fast ones for classification and high volume.
Why: Price and latency differ many times over between tiers, so matching the model to the task is the biggest cost lever you have.

Watch out

Picking a model by leaderboard rank instead of testing a few candidates on your own examples.

Go deeper