Gradient Boosting with XGBoost

Learning from mistakes

Gradient boosting builds trees sequentially, each one correcting the errors of the last. Done carefully, it produces extremely accurate models — which is why XGBoost and LightGBM dominate tabular competitions.

import xgboost as xgb
model = xgb.XGBClassifier(
    n_estimators=400, learning_rate=0.05, max_depth=5, subsample=0.8)
model.fit(X_train, y_train)

Key knobs

• learning_rate — smaller is more accurate but slower; pair with more trees.
• max_depth — controls complexity; watch for overfitting.
• subsample / colsample — add randomness to generalize better.

Use early stopping on a validation set to pick the right number of trees automatically.