Autograd

How PyTorch computes gradients for you: it records operations on tensors that need gradients, then walks that record backwards.

requires_grad

What: A flag on a tensor that tells autograd to track operations on it. Model parameters have it set; ordinary data tensors do not.
Why: Only tracked tensors get a .grad, and tracking costs memory, so it is on for what you train and off for everything else.

Python

w = torch.randn(3, requires_grad=True)
x = torch.tensor([1.0, 2.0, 3.0])
y = (w * x).sum()
y.requires_grad

Watch out

torch.randn(3, requires_grad=True).cuda() is the result of an operation, not a leaf, so it gets no .grad; create it on the device instead.

The computation graph

Also called dynamic graph, grad_fn

What: As tracked operations run, autograd builds a graph linking each result to its inputs through a grad_fn. It is rebuilt on every forward pass.
Why: Because the graph is dynamic, ordinary Python control flow (loops, ifs) works inside models.

Watch out

Keeping tensors like the loss in a Python list across steps keeps their whole graph alive; store loss.item() instead.

backward and .grad

What: Calling .backward() on a scalar computes the gradient of it with respect to every leaf tensor that requires grad, and adds the result into each tensor's .grad.
Why: This is how the loss tells each weight which way to move.

Python

w = torch.tensor([2.0, -1.0], requires_grad=True)
x = torch.tensor([3.0, 4.0])
loss = ((w * x).sum() - 1) ** 2
loss.backward()
w.grad

Watch out

Gradients accumulate: without optimizer.zero_grad() each step adds to the previous gradients.

no_grad and inference_mode

What: torch.no_grad() turns off graph recording inside its block. torch.inference_mode() does the same with extra optimisations for code that never needs gradients.
Why: Evaluation and prediction use less memory and run faster without the graph.

Python

model.eval()
with torch.inference_mode():
    preds = model(batch).argmax(dim=1)

with torch.no_grad():
    for p in model.parameters():
        p.clamp_(-1, 1)

Watch out

Tensors created under inference_mode cannot later be used in autograd; use no_grad if they might be.

detach

What: tensor.detach() returns a tensor that shares the data but is cut off from the graph, so no gradient flows through it.
Why: Logging values, stopping gradients into a target network, or converting to NumPy with .detach().cpu().numpy().

Python

features = encoder(x)
frozen = features.detach()
value = loss.detach().item()
arr = preds.detach().cpu().numpy()

Watch out

Changing a detached tensor in place also changes the original, because the memory is shared.

In-place operations

Also called add_, trailing underscore

What: Methods ending in an underscore, such as add_ and zero_, change a tensor in place instead of returning a new one.
Why: They save memory, which is why optimisers use them to update weights.

Watch out

An in-place change to a value autograd saved for backward raises "modified by an inplace operation"; use the out-of-place version.

Go deeper