One split can fool you

A single train/test split gives you one estimate of performance — and that estimate is at the mercy of which rows happened to land in the test set. Get an easy slice and you'll be over-optimistic; an unlucky slice and you'll discard a good model. For smaller datasets especially, one number isn't enough to trust.

Cross-validation

k-fold cross-validation fixes this by splitting the data into k parts (folds), then training k times — each time holding out a different fold as the test set and training on the rest. Average the k scores and you get a far more stable, harder-to-fool estimate, plus a sense of how variable performance is.

from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, X_train, y_train, cv=5)
print(scores.mean(), scores.std())

A solid mean and a low spread is what you want; a high mean with a huge spread means the model is unreliable.

Tuning hyperparameters

Models have hyperparameters — settings you choose rather than learn, like tree depth or regularization strength. You tune them by trying combinations and scoring each with cross-validation (grid search or random search automate this). Crucially, do all tuning on training and validation data and keep the test set untouched until the very end — otherwise you overfit to your own evaluation.

Putting it together

A realistic workflow: split off a test set, use cross-validation on the rest to engineer features and tune models, pick the best, then score it once on the test set for an honest final number.

Don't trust a model on the strength of one lucky split. Cross-validation turns a single noisy guess into a stable estimate — and a stable estimate is what lets you choose between models with confidence.

Try this: Run cross_val_score with cv=5 and look at the five numbers, not just the average. The spread between them is a humility check: if they swing wildly, your single-split "result" was mostly luck.