Building a model is just one piece of the puzzle. Getting ML working in production involves data pipelines, monitoring, retraining, and infrastructure.

Let's walk through the complete lifecycle of a real ML system.

The Full ML Lifecycle

Most tutorials stop at "train a model." Real-world ML continues far beyond:

1. Problem Definition
2. Data Collection
3. Data Preparation
4. Feature Engineering
5. Model Training
6. Model Evaluation
7. Model Deployment
8. Monitoring & Maintenance
9. Iteration & Improvement

Each stage is critical. Weak links anywhere break the entire system.

Stage 1: Problem Definition

Before writing any code, answer:

What exactly are you predicting?
- Vague: "Improve customer satisfaction"
- Specific: "Predict which customers will churn within 30 days"

What's the business impact?
- If 10% more accurate: Save $500K/year
- If 10% faster: Enable real-time personalization
- If interpretable: Meet regulatory requirements

What's your success metric?
- Not just model accuracy
- Business KPI: Revenue, retention, cost savings

What are the constraints?
- Latency: Must respond in <100ms
- Cost: Can't spend more than $X on compute
- Fairness: Must not discriminate by protected attributes
- Interpretability: Need to explain predictions to users

Is ML even necessary?
Sometimes simple rules work better:
- "If no purchase in 90 days, send email" might outperform a complex ML model
- Always try simple baselines first

Stage 2: Data Collection

ML is data-hungry. Where does it come from?

Internal databases:
- Transaction logs
- User behavior data
- Historical records
- Sensor readings

External sources:
- Public datasets
- Purchased data
- APIs (weather, maps, demographics)
- Web scraping (with permission)

Data quality questions:
- Is it representative of production scenarios?
- Does it have the features you need?
- Is it labeled (for supervised learning)?
- How much do you have? (More is usually better)

Common challenges:
- Insufficient data (need thousands to millions of examples)
- Biased sampling (only logs successful transactions, not failures)
- Class imbalance (99% normal, 1% fraud)
- Missing labels (have data but no ground truth)

Cold start problem:
If building a recommendation system, you need user behavior data. But users haven't used the system yet! Solutions: Start with rules, collect initial data, then introduce ML gradually.

Stage 3: Data Preparation

Raw data is messy. Clean it up:

Handle Missing Values
- Strategy 1: Remove rows with missing data
- Strategy 2: Fill with mean/median/mode
- Strategy 3: Use a model to predict missing values
- Strategy 4: Create a "missing" indicator feature

Choice depends on why data is missing and how much is missing.

Remove Duplicates
- Exact duplicates: Delete
- Near-duplicates: Requires judgment (Are two customer records the same person?)

Fix Inconsistencies
- Date formats: "2024-01-15" vs "01/15/2024" vs "Jan 15, 2024"
- Units: Miles vs kilometers, dollars vs cents
- Typos: "Califronia" → "California"
- Case sensitivity: "apple" vs "Apple" vs "APPLE"

Handle Outliers
- Detect: Values far from the mean (>3 standard deviations)
- Decide: Are they errors or genuine extreme cases?
- Action: Remove, cap, or keep

Example: Income of $10,000,000,000 is likely a data entry error. Income of $1,000,000 might be real.

Data validation:
- Check ranges: Age between 0-120, prices positive
- Check consistency: Start date before end date
- Check distributions: Sudden spikes might indicate errors

This stage often takes 60-80% of total project time!

Stage 4: Feature Engineering

Raw features are rarely optimal. Create better ones:

Derived Features
- From "Date of Birth" → "Age"
- From "Transaction Date" → "Day of Week", "Month", "Is Holiday"
- From "Product Price" and "Quantity" → "Total Purchase Value"

Aggregations
- Customer "Number of Purchases" in last 30/90/365 days
- Product "Average Rating" across all reviews
- Website "Session Duration" per visit

Interactions
- "Income × Credit Score" might predict loan default better than either alone
- "Temperature × Humidity" captures the "feels like" temperature

Binning
- Continuous age (0-100) → Categories (Child, Teen, Adult, Senior)
- Helps models find patterns: discounts work for Seniors, not Adults

Encoding Categorical Variables
- One-hot encoding: "Color" → separate binary columns for Red, Blue, Green
- Ordinal encoding: "Size" → Small=1, Medium=2, Large=3
- Target encoding: Encode category by average target value in training data

Text Features
- Bag of words: Count frequency of each word
- TF-IDF: Weight words by importance
- Embeddings: Dense vector representations (advanced)

Domain Knowledge
This is where expertise matters. A domain expert can create features a data scientist might miss.

Example: Predicting house prices
- Data scientist: Use size, bedrooms, bathrooms
- Real estate expert: Add "Distance to good schools", "Neighborhood walkability score", "Days since last similar sale"

Better features > fancier models

Stage 5: Model Training

Now the fun part—but don't skip the previous stages!

Training process:

1. Split data: Train (70%), Validation (15%), Test (15%)

2. Choose model family
- Start simple: Logistic regression, decision trees
- Try ensemble methods: Random forests, gradient boosting
- Consider deep learning: Neural networks (if you have lots of data)

3. Train initial model
Fit model on training data

4. Tune hyperparameters
- Learning rate
- Regularization strength
- Tree depth
- Number of layers
Use validation set to choose best values

5. Prevent overfitting
- Cross-validation
- Early stopping
- Regularization
- Dropout (for neural networks)

6. Ensemble if needed
Combine multiple models:
- Voting: Majority vote from 5 different models
- Stacking: Use one model to combine predictions from others
- Boosting: Sequentially train models to fix previous errors

Often gives 2-5% accuracy boost

7. Final evaluation on test set
Only after everything is finalized
This is your true performance estimate

Stage 6: Model Evaluation

Beyond accuracy:

Business Metrics
- Will this actually increase revenue?
- Does it reduce costs?
- Does it improve user experience?

Compare to baselines:
- Random guessing
- Simple rules ("Always predict most common class")
- Current system (if replacing an existing one)
- Human performance (if available)

Your model must beat these to be worthwhile.

Fairness Checks
Test performance across different subgroups:
- Does it work equally well for all demographics?
- Are there disparate impacts?
- Could it discriminate unfairly?

Error Analysis
Look at mistakes:
- What types of examples does it get wrong?
- Are there patterns?
- Can you add features to fix these?

Example: Fraud detector misses sophisticated fraud but catches obvious cases
→ Add features about transaction patterns, not just amounts

Robustness Testing
- How does it handle edge cases?
- What about corrupted or adversarial inputs?
- Does performance degrade gracefully?

Latency & Resource Requirements
- Can it make predictions fast enough?
- What hardware does it need?
- What's the cost per prediction?

A 99% accurate model that takes 10 seconds per prediction might be useless if you need real-time results.

Stage 7: Model Deployment

Getting your model into production:

Deployment Strategies

Option 1: Batch Predictions
- Run model periodically (daily, weekly)
- Pre-compute predictions for all users
- Store in database
- Serve pre-computed predictions

Good for: Email campaigns, report generation

Option 2: Real-Time API
- Model runs on-demand
- User request → Model prediction → Immediate response
- Need low latency (<100ms often)

Good for: Fraud detection, recommendations, chatbots

Option 3