A/B Testing for ML Features

Offline ≠ online

A model that scores better offline doesn't always help users. A/B testing routes some users to the new model and compares real outcomes (clicks, conversions, retention).

Do it right

• Randomize assignment and keep groups comparable.
• Pick one primary metric tied to real value.
• Size the test for enough power, and run it long enough.
• Watch guardrail metrics (latency, complaints) for regressions.

The payoff

A/B testing is the ground truth that closes the loop between model quality and business impact. Trust it over offline scores when they disagree.