The three dials of learning

The training loop has three components you actually choose, and understanding them turns "the model won't learn" from a mystery into a checklist. They are the activation function, the loss function, and the optimizer — the squash, the scorecard, and the stepper.

Activation functions: the squash

Each neuron's non-linearity is its activation. A few dominate:

• ReLU — outputs the input if positive, else zero. Simple, fast, and the default for hidden layers in most networks.
• Sigmoid — squashes to between 0 and 1, useful for a probability at the output.
• Softmax — turns a row of numbers into probabilities that sum to 1, for picking one of several classes.

For hidden layers, when in doubt, ReLU. The choice mostly matters at the output, where it must match the task.

Loss functions: the scorecard

The loss is the single number that says how wrong the model is — and it defines what "good" even means, so it must fit the task:

• Cross-entropy for classification (which category?).
• Mean squared error for regression (predict a number).

The loss is what backprop drives toward zero, so choosing the wrong one quietly trains the wrong behavior.

Optimizers: the stepper

The optimizer decides how to turn gradients into weight updates. Plain gradient descent works, but smarter optimizers converge faster and more reliably:

• SGD — the classic; steady and well-understood.
• Adam — adapts the step size per weight and is the practical default for most deep learning.

The learning rate is the single most important knob across all of them: too high and training diverges, too low and it barely moves.

Activation, loss, optimizer — squash, scorecard, stepper. When a network won't learn, check these three (and the learning rate) before you suspect anything fancier.

Try this: For a task you can imagine — sorting photos into five categories — name the right output activation (softmax), the right loss (cross-entropy), and a safe optimizer (Adam). Matching the three to the task is a skill you'll reuse on every model you build.