Never grade on the training set

Imagine a student who gets the exam questions in advance, memorizes the answers, and aces the test. Did they learn the subject? You have no idea — and that's exactly what happens if you evaluate a model on the same data it trained on. It can score beautifully by memorizing while being useless on anything new. So the first rule of honest ML is: test on data the model has never seen.

The three splits

We carve the data into parts with different jobs:

• Training set (typically 60–80%) — the model learns from this.
• Validation set — used during development to tune settings and compare model choices.
• Test set — locked away and touched once, at the very end, to estimate real-world performance.

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

The validation set matters because the moment you use the test set to make decisions, it stops being unseen — you've started fitting to it indirectly. Keep it sacred.

Leakage: the silent killer

The subtle failure here is data leakage — when information from the test set sneaks into training. Classic examples: scaling your features using statistics computed over the whole dataset, or including a column that secretly encodes the answer. Leakage produces dazzling test scores that evaporate in production. Always split first, then fit any preprocessing on the training data only.

Your test score is only as honest as your split. Memorization looks identical to learning until you check on data the model has never touched — so guard that data carefully.

Try this: Next time you see an ML result that sounds too good ("99% accurate!"), ask one question: tested on what? If the answer is "the data it trained on," the number means nothing. That single reflex makes you a sharper consumer of every AI claim.