"My model has 95% accuracy!" sounds impressive. But what if 95% of your data belongs to one class?

Choosing the right evaluation metric is just as important as building the model itself. The wrong metric can make a terrible model look good—or a good model look terrible.

Why Accuracy Can Be Misleading

Imagine a medical test for a rare disease that affects 1% of the population.

Terrible Model: Always predicts "No disease"
- Result: 99% accurate!
- Why: It's right 99 times out of 100 (when person is healthy)
- Problem: It never catches the actual disease

This is called the "class imbalance problem." When one class vastly outnumbers the other, accuracy becomes meaningless.

You need metrics that account for this.

The Confusion Matrix: Your Foundation

Every classification metric builds from the confusion matrix.

Example: Email spam detection (1,000 emails tested)

Predicted: Not Spam | Predicted: Spam
Actual: Not Spam 850 | 50
Actual: Spam 30 | 70

Breaking this down:

True Negatives (TN): 850
- Correctly identified as not spam
- Model is correct, prediction is negative

False Positives (FP): 50
- Incorrectly flagged as spam (false alarm)
- Model is wrong, prediction is positive
- Type I error

False Negatives (FN): 30
- Spam that got through (missed)
- Model is wrong, prediction is negative
- Type II error

True Positives (TP): 70
- Correctly identified as spam
- Model is correct, prediction is positive

All evaluation metrics derive from these four numbers.

Key Metrics for Classification

Accuracy
Formula: (TP + TN) / (TP + TN + FP + FN)
Our example: (70 + 850) / 1000 = 92%

When to use: Balanced datasets where all classes matter equally
When NOT to use: Imbalanced data, or when different errors have different costs

Precision
Formula: TP / (TP + FP)
Our example: 70 / (70 + 50) = 58%

Interpretation: Of all emails marked as spam, 58% actually were spam.

When to use: When false positives are costly
Example: Medical diagnosis—you don't want to tell healthy people they're sick

Recall (Sensitivity)
Formula: TP / (TP + FN)
Our example: 70 / (70 + 30) = 70%

Interpretation: Of all actual spam emails, we caught 70%.

When to use: When false negatives are costly
Example: Fraud detection—missing fraud is worse than false alarms

F1 Score
Formula: 2 × (Precision × Recall) / (Precision + Recall)
Our example: 2 × (0.58 × 0.70) / (0.58 + 0.70) = 63%

Interpretation: Harmonic mean of precision and recall

When to use: When you need a single metric that balances both precision and recall

The Precision-Recall Tradeoff

You can't maximize both simultaneously. There's always a tradeoff.

Increase recall (catch more spam):
- Mark more emails as spam
- Catch 90% of spam (FN decreases)
- But also flag more legitimate emails (FP increases)
- Precision drops to 40%

Increase precision (fewer false alarms):
- Only mark obvious spam
- 95% of flagged emails are actually spam (FP decreases)
- But you miss half the spam (FN increases)
- Recall drops to 50%

Your choice depends on business priorities:

Customer-facing chatbot: High precision (don't give wrong answers)
Security system: High recall (don't miss threats)
General classifier: Balance with F1 score

Metrics for Regression Problems

When predicting continuous values (prices, temperatures, distances):

Mean Absolute Error (MAE)
Formula: Average of |actual - predicted|

Example house prices:
- House 1: Actual $300k, Predicted $320k → Error: $20k
- House 2: Actual $250k, Predicted $230k → Error: $20k
- House 3: Actual $400k, Predicted $390k → Error: $10k
- MAE: ($20k + $20k + $10k) / 3 = $16.7k

Interpretation: On average, predictions are off by $16,700

Pros: Easy to understand, same units as your target
Cons: Treats all errors equally (small and large)

Mean Squared Error (MSE)
Formula: Average of (actual - predicted)²

Same example:
- MSE: ($20k² + $20k² + $10k²) / 3 = $283M

Interpretation: Emphasizes larger errors more (squared!)

Pros: Penalizes large errors heavily
Cons: Units are squared (harder to interpret)

Root Mean Squared Error (RMSE)
Formula: √MSE
Our example: √$283M = $16.8k

Interpretation: Back to original units, but still emphasizes large errors

When to use: When large errors are particularly bad (safety-critical systems)

R² Score (Coefficient of Determination)
Formula: 1 - (sum of squared errors / total variance)
Range: 0 to 1 (sometimes negative if model is terrible)

Our example: R² = 0.85

Interpretation: Model explains 85% of variance in house prices

When to use: Comparing models, understanding explanatory power
Perfect model: R² = 1
Baseline (mean prediction): R² = 0
Worse than baseline: R² < 0

ROC Curve and AUC

For binary classification, there's a powerful visual tool:

ROC Curve (Receiver Operating Characteristic)
- X-axis: False Positive Rate
- Y-axis: True Positive Rate (Recall)
- Shows performance at all decision thresholds

Example: Your model outputs probabilities (0.0 to 1.0)
- Threshold 0.5: Classify as spam if probability > 0.5
- Threshold 0.7: Classify as spam if probability > 0.7
- Threshold 0.3: Classify as spam if probability > 0.3

Each threshold gives different TPR and FPR. Plot them all.

AUC (Area Under Curve)
- Single number summarizing the ROC curve
- Range: 0.5 (random guessing) to 1.0 (perfect)
- Our spam detector: AUC = 0.88 (pretty good!)

When to use:
- Comparing models with a single metric
- When you're unsure about the best threshold
- When class imbalance exists

Perfect classifier: AUC = 1.0
Random classifier: AUC = 0.5
Worse than random: AUC < 0.5

Choosing the Right Metric: Decision Tree

Ask yourself:

Is it classification or regression?
→ Regression: Use MAE, RMSE, or R²
→ Classification: Continue...

Is the dataset balanced?
→ Yes: Accuracy is fine
→ No: Continue...

What's more costly?
→ False positives worse: Optimize for precision
→ False negatives worse: Optimize for recall
→ Both matter equally: Use F1 score

Do you need a single threshold-independent metric?
→ Yes: Use AUC
→ No: Stick with precision/recall at your chosen threshold

Example decisions:

Credit card fraud: Recall (catch all fraud)
Email spam: Precision (don't block legitimate emails)
Cancer screening: Recall (don't miss any cases)
Job applicant screening: Precision (only interview qualified candidates)

Multi-Class Classification Metrics

When you have more than 2 classes (e.g., classifying animals: cat, dog, bird):

Accuracy still works:
- Correct predictions / Total predictions

But for detailed analysis:

Macro-averaged metrics:
- Calculate metric for each class separately
- Average them (treats all classes equally)

Weighted-averaged metrics:
- Calculate metric for each class
- Weight by class frequency
- Better for imbalanced datasets

Micro-averaged metrics:
- Pool all classes together
- Calculate one overall metric

Example confusion matrix for 3 classes:
Pred: Cat | Pred: Dog | Pred: Bird
Actual: Cat 85 | 10 | 5
Actual: Dog 12 | 80 | 8
Actual: Bird 3 | 7 | 90

You'd calculate precision and recall for each class, then average.

Business Metrics vs. Model Metrics

Remember: Model metrics (F1, AUC) are proxies for business goals.

What actually matters:

E-commerce recommendations:
- Model metric: Precision@K
- Business metric: Revenue from recommended products

Fraud detection:
- Model metric: Recall
- Business metric: $ amount of fraud prevented vs. false alarm costs

Medical diagnosis:
- Model metric: Sensitivity/Specificity
- Business metric: Lives saved, quality of life, treatment costs

Always tie your model metrics to business KPIs. A model with 95% accuracy that generates no business value is worthless.

Common Mistakes to Avoid

1. Using accuracy on imbalanced data
2. Optimizing for precision when recall matters (or vice versa)
3. Not setting a baseline (compare to simple rules or existing systems)
4. Ignoring business context when choosing metrics
5. Reporting only a single metric (show multiple perspectives)
6. Not validating on truly unseen test data
7. Cherry-picking metrics that make your model look good

Best practice: Report multiple metrics and explain tradeoffs.

Summary Table

Problem Type | Primary Metrics | When to Use
-------------|----------------|-------------
Balanced Classification | Accuracy, F1 | Classes roughly equal size
Imbalanced Classification | Precision, Recall, AUC | One class much rarer
Ranking/Recommendations | Precision@K, MAP | Order matters
Regression | MAE, RMSE, R² | Predicting continuous values
Multi-class | Macro/Micro F1 | More than 2 classes

Next up: We'll move to Section 3 where we explore unsupervised learning techniques and when to use them.