The Illustrated Transformer

What it covers

The article walks through the original transformer from the 2017 paper, one diagram at a time:

• The high-level encoder and decoder structure.
• How word vectors flow through each component as tensors.
• Self-attention, first as a concept with queries, keys and values, then as matrix calculations.
• Multi-headed attention and why several heads help.
• Positional encoding, residual connections and layer normalisation.
• The decoder, the final linear and softmax layer, and how training and the loss work.

It was first published in June 2018, clarified in 2020, and has been translated into eleven languages. It has been used in university courses at places such as Stanford, MIT and CMU.

Why it works when other explanations do not

The original paper is short and assumes a lot. Many secondary explanations either restate the equations or wave their hands. This one shows the actual shape of the data at each step, so you can see that attention is each word building a weighted mix of the others, and that a head is simply one learned way of choosing those weights. For people who think in structures rather than algebra, that is the missing piece.

Before you read it

You will get more from it if you know roughly what a neural network layer, a vector and a matrix multiplication are. If those are shaky, watch the first chapters of the 3Blue1Brown neural network series first. You do not need to have read the original paper.

Set aside about an hour, with pen and paper. Work through the self-attention matrix section slowly, and try to sketch the diagram for a three-word sentence yourself.

Is it still current?

The article describes the original encoder and decoder transformer. Modern language models are decoder-only and use refinements such as rotary position embeddings and grouped or multi-query attention. The core idea is unchanged, which is why the article remains the best starting point, but it is not a description of how current models are built in detail.

In 2025 the author noted that an updated and expanded version, covering those newer variations, appears as chapter 3 of the book Hands-On Large Language Models, and that a free short course with animations was released.

What to read after

• The same author's follow-up posts, such as The Illustrated GPT-2, which shows the decoder-only version.
• The transformer and attention chapters of the 3Blue1Brown series for an animated view of the same ideas.
• Andrej Karpathy's lecture building a small GPT in code. Diagram first, then implementation, is a very effective order.

Common questions

Do I need to know the maths? Basic matrix multiplication helps with one section; the rest is visual.

Is it enough to understand ChatGPT-style models? It gives you the core mechanism. Pair it with a decoder-only explanation for the modern picture.

Best for: anyone who wants to finally understand attention and the transformer before working with language models.