harsh040506/self-healing-models

Self-Healing Models — Drift Detection and Autonomous Recovery

Provides the complete framework for detecting statistical distribution shifts in production ML systems and autonomously responding with shadow retraining and zero-downtime model updates. Implements the "Self-Healing Model" concept where concept drift triggers an end-to-end recovery cycle without manual intervention.

Concept Drift Taxonomy

| Drift Type | Description | Detection Method | |---|---|---| | Sudden drift | Abrupt change at a specific time point (e.g., system change, world event) | Immediate KS test comparison | | Gradual drift | Slow, continuous shift (e.g., user behavior evolution) | Rolling window PSI trend | | Recurrent drift | Seasonal or cyclic patterns reappear (e.g., holiday behavior) | Season-aware baseline comparison | | Incremental drift | Monotonic directional drift over time | Linear trend test on PSI | | Feature drift | Input distribution changes but relationship to target is stable | Input-only KS/PSI (re-calibrate, don't retrain) | | Label drift | Relationship between features and target changes (true concept drift) | Requires label monitoring + performance tracking |

Statistical Tests

Kolmogorov-Smirnov (KS) Test for continuous features:

• D = max|F_ref(x) − F_current(x)| (maximum CDF difference)
• p-value < 0.05 → statistically significant drift
• Sensitive to any shape of distribution change (shift, scale, shape)

Chi-Square Test for categorical features:

• χ² = Σ (O_i − E_i)² / E_i
• O_i = observed frequency in current period; E_i = expected frequency from reference
• df = k - 1 (k = number of categories); p-value < 0.05 → drift

Population Stability Index (PSI) for magnitude quantification:

• PSI = Σ (P_current_i − P_reference_i) · ln(P_current_i / P_reference_i)
• PSI < 0.1: negligible shift — no action
• 0.1 ≤ PSI < 0.2: moderate shift — increase monitoring frequency, alert team
• PSI ≥ 0.2: major shift — trigger retraining pipeline

Drift aggregation rule (avoids false positives from individual feature noise):

• Trigger retraining only when: ≥ 3 features have KS p < 0.05 AND ≥ 1 feature has PSI ≥ 0.2, OR performance metric degrades > 5% relative.

See references/drift-detection.md for complete test implementations, window size selection, and rolling-window monitoring patterns.

Self-Healing Pipeline Sequence

1. Drift Detected (KS + PSI thresholds exceeded)
          ↓
2. Shadow Training Triggered (new data distribution)
          ↓
3. Challenger Model Evaluated vs. Champion
          ↓
4. Statistical Significance Test (paired t-test, p < 0.05)
          ↓
5. Blue-Green Deployment (10% canary → 100% cutover)
          ↓
6. Champion model archived (24-hour rollback window)

Shadow Training

Shadow training runs the full training pipeline on the drifted data distribution without disrupting the production model:

• Isolate: use the current data window (typically last 30–90 days) as the new training distribution
• Architecture: reuse the same architecture unless drift analysis suggests a fundamentally different data signature
• HPO: reuse the last best hyperparameters; re-run HPO only if drift severity is "critical" (PSI ≥ 0.5)
• Parallelism: shadow model trains on a separate compute partition — no production impact
• Hardware: prefer spot instances for cost efficiency; configure checkpointing for preemption resilience

Champion-Challenger Evaluation

Before any deployment, the challenger must beat the champion on the current data distribution:

1. Set aside 20% of the current data window as a holdout evaluation set (after shadow training).
2. Evaluate both champion and challenger on the holdout set.
3. Run a paired t-test (or permutation test) on per-sample losses:
   • H₀: challenger performance = champion performance
   • Reject H₀ if p < 0.05 → challenger is statistically better
4. Require: challenger primary metric improvement ≥ 2% relative AND p < 0.05.
5. Check all quality gates: latency SLA, fairness thresholds, no regression on reference distribution.

See references/deployment-strategies.md for blue-green, canary, and shadow traffic routing patterns with Kubernetes ingress configurations.

Rollback Protocol

If post-deployment monitoring detects regression within 24 hours:

1. Immediately route 100% traffic back to the previous champion (now in "green" slot).
2. Log the regression with full diagnostics: which metrics degraded, by how much.
3. File an incident report and increase shadow training data window (e.g., 30 → 90 days).
4. Re-evaluate whether drift was correctly classified (feature drift vs. true label drift).