harsh040506/mlops-pipeline

MLOps Pipeline — End-to-End ML Lifecycle Management

Provides systematic guidance for operationalizing machine learning models: from data ingestion and feature engineering through experiment tracking, deployment, and production monitoring. Directly addresses the "Hidden Technical Debt in ML Systems" (Sculley et al.) by making the surrounding infrastructure as rigorous as the model itself.

The MLOps Stack

Data Sources → Feature Store → Training Pipeline → Model Registry → Serving Infrastructure → Monitoring

Each stage must be reproducible, versioned, and auditable.

Data Ingestion and Quality

Schema inference is the first gate:

• Detect column types automatically: numeric, categorical, datetime, text, target
• Compute: null rates, cardinality, value distributions, duplicate rows
• Block pipeline on: > 20% missing values in target, > 5% duplicate rows, schema drift from expected

Outlier detection (run before any feature engineering):

• Numeric: IQR method (flag outside 1.5×IQR as outlier, > 3×IQR as extreme outlier)
• Categorical: flag categories with < 0.1% frequency (likely data entry errors)

See references/data-engineering.md for full data quality checks, validation schemas, and data contract patterns.

Feature Engineering

Normalization (always log transformation params for inference parity):

• Standard scaling: x' = (x − μ) / σ — save μ, σ from training set; apply at inference
• Min-max: x' = (x − x_min) / (x_max − x_min)
• Log1p: for right-skewed features (income, counts, prices)

Encoding:

• Low-cardinality categorical (< 30): one-hot encoding
• High-cardinality categorical (30–1000): target encoding (mean of target per category)
• Very high cardinality (> 1000): entity embeddings (learned dense representations)

Temporal features: extract as cyclical sin/cos pairs to preserve periodicity:

• hour_sin = sin(2π · hour / 24), hour_cos = cos(2π · hour / 24)

See references/feature-stores.md for feature store architecture, point-in-time correct joins, and online/offline serving patterns.

Experiment Tracking

Every training run must record: | Artifact | Purpose | |---|---| | Git commit SHA | Code reproducibility | | Dataset hash (MD5/SHA256) | Data reproducibility | | Full hyperparameter config | Experiment reproducibility | | Random seed | Run reproducibility | | Environment (Python + library versions) | Dependency reproducibility |

Use MLflow or Weights & Biases for automatic artifact logging.

Model Deployment Patterns

Dev: Local FastAPI endpoint for integration testing Staging: Docker container → Kubernetes (namespace: staging) + smoke tests Production: Blue-green or canary deployment (see self-healing-models skill)

Model serialization formats:

• model.pt — PyTorch, for fine-tuning and retraining
• model.onnx — runtime-agnostic, for cross-platform serving
• model.pkl — Scikit-learn pipeline including preprocessing steps

Always include the preprocessing pipeline in the serialized model artifact to prevent training-serving skew.

See references/monitoring.md for production monitoring setup, alerting rules, and dashboard templates.

Quality Gates

Before promoting any model to production:

1. Primary metric exceeds configured threshold
2. Fairness checks pass (if sensitive attributes present)
3. Latency P99 < configured SLA (e.g., 50ms)
4. No statistically significant regression vs. current champion
5. Model card completed and reviewed