Running models with transformers

The transformers library: from a one-line pipeline to loading the tokeniser and model yourself and generating text.

pipeline

What: pipeline(task, model=...) bundles preprocessing, the model and postprocessing for a task such as text generation, classification, summarisation or speech recognition.
Why: The fastest way to try a model and a good default for simple inference.

Python

from transformers import pipeline

sst2 = "distilbert/distilbert-base-uncased-finetuned-sst-2-english"
clf = pipeline("text-classification", model=sst2)
clf("The update fixed every crash I had.")

asr = pipeline("automatic-speech-recognition",
               model="openai/whisper-small")
asr("meeting.wav")["text"]

Watch out

Leaving out model=: the task default may change between versions and a warning says so.

Auto classes

Also called AutoModel, AutoModelForCausalLM, from_pretrained

What: Auto classes read a repo's config and build the right architecture. The suffix picks the head: AutoModelForCausalLM for text generation, AutoModelForSequenceClassification for labels, plain AutoModel for hidden states.
Why: The same code loads thousands of models; changing the repo id is enough.

Python

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Qwen/Qwen3-0.6B"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo, dtype="auto", device_map="auto"
)

Watch out

Loading a base model with a classification head warns that some weights are newly initialised: that head is random until you fine-tune it.

dtype and device_map

What: dtype="auto" loads weights in the precision they were saved in, often bfloat16, instead of float32. device_map="auto" places layers across available GPUs and the CPU.
Why: Loading in full float32 doubles memory; automatic placement lets a model larger than one GPU still run.

Watch out

Older code passes torch_dtype; current releases name the argument dtype.

generate

Also called max_new_tokens, temperature, sampling

What: model.generate produces tokens one at a time. max_new_tokens caps the length, do_sample with temperature and top_p adds randomness, and greedy decoding picks the top token each step.
Why: Decoding settings change output quality, variety and cost as much as the model does.

Python

inputs = tok("The capital of Japan is", return_tensors="pt")
inputs = inputs.to(model.device)
out = model.generate(**inputs, max_new_tokens=30,
                     do_sample=True, temperature=0.7, top_p=0.9)
new = out[0, inputs["input_ids"].shape[1]:]
print(tok.decode(new, skip_special_tokens=True))

Watch out

Decoding the whole output and wondering why the prompt appears again; slice off the input tokens first.

Chat templates

Also called apply_chat_template

What: Each chat model expects messages formatted with its own special tokens. apply_chat_template turns a list of role and content messages into exactly that format.
Why: A chat model given plain text or the wrong format answers poorly or rambles.

Python

messages = [
    {"role": "system", "content": "Answer in one sentence."},
    {"role": "user", "content": "What is LoRA?"},
]
inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True,
    return_tensors="pt", return_dict=True,
).to(model.device)
out = model.generate(**inputs, max_new_tokens=100)

Watch out

Forgetting add_generation_prompt=True, so the model continues the user turn instead of replying.

Model outputs and heads

Also called logits, last_hidden_state

What: Calling a model returns an output object: logits for models with a task head, last_hidden_state for a bare AutoModel, and loss when you pass labels.
Why: Knowing which field to read is how you turn a forward pass into predictions or features.

Python

import torch

enc = tok(["great film", "dull film"], padding=True,
          return_tensors="pt")
with torch.inference_mode():
    logits = clf_model(**enc).logits
probs = logits.softmax(dim=-1)
id2label = clf_model.config.id2label
labels = [id2label[i] for i in probs.argmax(-1).tolist()]

safetensors

What: The default weight format on the Hub: a simple file of tensors with a JSON header, which loads fast and cannot run code.
Why: Unlike pickle-based .bin files, loading an untrusted safetensors file is safe.

Watch out

Passing trust_remote_code=True runs Python from the repo; only do it for repos you have read and trust.

Go deeper