sawrus/areas/software/mlops/skills/model-monitoring

Skill: Model Monitoring

When to load

When setting up monitoring for a deployed model or responding to drift alerts.

Monitoring Dimensions

1. Operational health
   - Latency: p50, p95, p99
   - Error rate: prediction failures, input validation failures

2. Data drift (vs training baseline)
   - PSI (Population Stability Index) per feature
   - PSI > 0.2 = significant shift → retrain likely needed

3. Model quality (when labels available)
   - Accuracy metrics after ground truth arrives
   - Business outcome correlation

PSI Drift Detection

def calculate_psi(expected: np.ndarray, actual: np.ndarray, buckets: int = 10) -> float:
    """PSI < 0.1: stable. 0.1-0.2: monitor. > 0.2: retrain."""
    breakpoints = np.percentile(expected, np.linspace(0, 100, buckets + 1))
    exp_counts = np.clip(np.histogram(expected, breakpoints)[0] / len(expected), 1e-4, None)
    act_counts = np.clip(np.histogram(actual, breakpoints)[0] / len(actual), 1e-4, None)
    return np.sum((act_counts - exp_counts) * np.log(act_counts / exp_counts))