The classic trap

Overfitting is the central failure of machine learning, and once you can spot it you'll see it everywhere. It happens when a model learns the training data too well — absorbing not just the real pattern but the random noise and quirks of those specific examples. It then performs brilliantly on what it has seen and falls apart on anything new. The tell-tale sign is a gap: high training score, low test score.

The opposite failure is underfitting — the model is too simple to capture the real pattern, so it does poorly on both training and test data. ML lives in the tension between these two.

A mental picture

Think of fitting a line through scattered points. A straight line might underfit — too rigid to follow the trend. A wildly wiggly curve that threads through every single point overfits: it nails the training dots but predicts nonsense between them. The sweet spot is the smooth curve that captures the trend and ignores the jitter.

Defenses that work

• More data — the single most reliable cure; noise averages out as examples pile up.
• Simpler models — fewer parameters means less room to memorize.
• Regularization — explicitly penalize complexity (L1/L2) so the model prefers simpler explanations.
• Cross-validation — evaluate across several splits for a stabler, harder-to-fool estimate.
• Early stopping — quit training once validation performance stops improving.

How you catch it

Watch training and validation scores together. When training keeps improving but validation plateaus or worsens, you've crossed into overfitting — stop or simplify.

A model that aces the training set proves nothing; it might just have a good memory. The only score that matters is on data it has never seen, and the gap between the two is your overfitting alarm.

Try this: Train a deliberately over-complex model on a small dataset and watch the gap appear — near-perfect training accuracy, mediocre test accuracy. Then add data or simplify the model and watch the gap shrink. Seeing the two numbers diverge and converge makes overfitting permanently intuitive.