Activation Functions, Demystified

Why non-linearity

Without an activation function, stacking layers just gives another linear function. Non-linear activations let networks learn curves, corners, and complex patterns.

The common ones

• ReLU — fast, simple, the default for hidden layers.
• GELU — smooth ReLU variant used in transformers.
• Sigmoid — squashes to (0,1), used for binary output.
• Softmax — turns scores into a probability distribution for multi-class output.

Rule of thumb

Use ReLU (or GELU) in hidden layers; choose the output activation to match your task (sigmoid/softmax for classification, none for regression).