PyTorch cheat sheet

1. 1TensorsThe n-dimensional array every PyTorch program is built on: how to create one, what it holds, how it is shaped and where it lives.7 concepts

   1. 1Creating tensorsA tensor is a typed, n-dimensional array.
   2. 2Data typesEvery tensor has one dtype.
   3. 3Shapes and reshapingshape lists the size of each dimension.
   4. 4BroadcastingWhen two tensors of different shapes meet in an operation, PyTorch lines up their shapes from the right and stretches size-one dimensions to match, without…
   5. 5Indexing and maskingTensors support NumPy-style slicing, integer index lists and boolean masks.
   6. 6DevicesA tensor lives on one device: the CPU, an NVIDIA GPU (cuda), Apple silicon (mps) or others.
   7. 7Reductions and matrix mathsReductions such as sum, mean, max and argmax take a dim to reduce along.

   7 in this chapterOpen this chapter
2. 2AutogradHow PyTorch computes gradients for you: it records operations on tensors that need gradients, then walks that record backwards.6 concepts

   1. 1requires_gradA flag on a tensor that tells autograd to track operations on it.
   2. 2The computation graphAs tracked operations run, autograd builds a graph linking each result to its inputs through a grad_fn.
   3. 3backward and .gradCalling .backward() on a scalar computes the gradient of it with respect to every leaf tensor that requires grad, and adds the result into each tensor's…
   4. 4no_grad and inference_modetorch.no_grad() turns off graph recording inside its block.
   5. 5detachtensor.detach() returns a tensor that shares the data but is cut off from the graph, so no gradient flows through it.
   6. 6In-place operationsMethods ending in an underscore, such as add_ and zero_, change a tensor in place instead of returning a new one.

   6 in this chapterOpen this chapter
3. Models with nn.ModuleHow to define a network: a class that owns its layers and says how data flows through them, plus the common building blocks.7 conceptsOpens with credits
4. Losses, optimisers and schedulersWhat the model minimises, the algorithm that updates the weights, and how the learning rate changes over time.7 conceptsOpens with credits
5. Datasets and DataLoadersHow data reaches the model: a Dataset knows how to fetch one example, and a DataLoader batches, shuffles and loads them in parallel.6 conceptsOpens with credits
6. Training, evaluation and checkpointsThe loop you write in every project, the evaluation loop beside it, and how to save and restore work.7 conceptsOpens with credits
7. Speed: GPUs, mixed precision and torch.compileThe switches that make the same model train faster and fit in less memory.7 conceptsOpens with credits
8. Debugging trainingHow to find out why a model will not run, will not learn or learns the wrong thing.6 conceptsOpens with credits
9. Distributed training and exportTraining on several GPUs, and turning a trained model into something another program can run.6 conceptsOpens with credits
10. 10Glossary, A to ZEvery concept in the sheet and the smaller terms it uses, in one list.83 terms

Tensors, autograd, nn.Module, training loops, data loading, GPUs, mixed precision, torch.compile, DDP and export in PyTorch.