Derivatives and gradients

A derivative tells you how fast a function's output changes when you nudge its input a little. For f(x) = x², the derivative is 2x. So at x = 3, a tiny increase in x raises f about six times as much.

A gradient is the same idea for a function with many inputs. It is a list of partial derivatives, one per input, and together they point in the direction where the output rises fastest. Training a model means adjusting its weights to make the loss, a score for how wrong it is, smaller.

The gradient of the loss for each weight says which way to move that weight and how strongly. Libraries work these out for you, so you rarely derive them by hand. But you must understand what a gradient means, or exploding and vanishing gradients will make no sense later.