Tensors

Creating tensors

Also called torch.tensor, torch.zeros, torch.randn

What

A tensor is a typed, n-dimensional array. torch.tensor copies Python data, while factories such as zeros, ones, arange and randn build one from a shape.

Why

Inputs, labels, weights and gradients are all tensors, so every step of training starts here.

import torch

x = torch.tensor([[1.0, 2.0], [3.0, 4.0]])
z = torch.zeros(3, 4)
r = torch.randn(2, 3)
steps = torch.arange(0, 10, 2)
like = torch.ones_like(x)
from_np = torch.from_numpy(np_array)

Data types

Also called dtype, float32, bfloat16, int64

What

Every tensor has one dtype. The defaults are float32 for floats and int64 for integers; float16 and bfloat16 halve memory, and bool holds masks.

Why

Losses and layers expect specific dtypes: class labels for cross entropy must be int64, and weights and inputs must match.

x = torch.tensor([1, 2, 3])
x.dtype
y = x.float()
h = y.to(torch.bfloat16)
labels = torch.tensor([0, 2, 1], dtype=torch.long)

Shapes and reshaping

Also called view, reshape, permute, squeeze, unsqueeze

What

shape lists the size of each dimension. reshape and view change the shape without changing the data, permute reorders dimensions, and unsqueeze or squeeze add or remove size-one dimensions.

Why

Most bugs in model code are shape bugs; images are usually (batch, channels, height, width) and sequences (batch, length, features).

x = torch.randn(8, 3, 32, 32)
x.shape
flat = x.reshape(8, -1)
hwc = x.permute(0, 2, 3, 1)
one = torch.randn(3, 32, 32).unsqueeze(0)
flat.shape, hwc.shape, one.shape

Broadcasting

What

When two tensors of different shapes meet in an operation, PyTorch lines up their shapes from the right and stretches size-one dimensions to match, without copying data.

Why

Adding a bias vector to every row or normalising every channel by its mean is one line instead of a loop.

x = torch.randn(64, 10)
bias = torch.randn(10)
y = x + bias
mean = x.mean(dim=0, keepdim=True)
centred = x - mean

Indexing and masking

Also called slicing, boolean mask, gather

What

Tensors support NumPy-style slicing, integer index lists and boolean masks. gather picks values along a dimension by index.

Why

Selecting a batch, the last time step of a sequence or the predicted class scores without Python loops.

x = torch.randn(4, 5)
first_row = x[0]
last_col = x[:, -1]
positives = x[x > 0]
picked = x[torch.tensor([0, 2])]
idx = torch.tensor([[1], [3], [0], [2]])
chosen = x.gather(1, idx)

Devices

Also called cuda, mps, .to(device)

What

A tensor lives on one device: the CPU, an NVIDIA GPU (cuda), Apple silicon (mps) or others. .to(device) moves it.

Why

Operations need all their tensors on the same device, and choosing the device once keeps the code portable.

if torch.cuda.is_available():
    device = torch.device("cuda")
elif torch.backends.mps.is_available():
    device = torch.device("mps")
else:
    device = torch.device("cpu")

x = torch.randn(2, 3, device=device)
model = model.to(device)

Reductions and matrix maths

Also called sum, mean, argmax, matmul, @

What

Reductions such as sum, mean, max and argmax take a dim to reduce along. @ or torch.matmul multiplies matrices, batching over leading dimensions.

Why

Accuracy, losses and every linear layer come down to these operations.

logits = torch.randn(32, 10)
preds = logits.argmax(dim=1)
acc = (preds == labels).float().mean()
w = torch.randn(10, 4)
out = logits @ w