Gradient Descent, Intuitively

Rolling downhill

Imagine the loss as a hilly landscape and your model's parameters as a position on it. Gradient descent repeatedly steps downhill — in the direction that most reduces the loss — until it reaches a valley.

The learning rate

The step size is the learning rate. Too large and you overshoot and diverge; too small and training crawls. It's the single most important hyperparameter to get roughly right.

Variants you'll hear about

SGD uses small batches for speed; Adam adapts the step size per parameter and is a strong default. They're all the same core idea: follow the gradient downhill.