qa-aman/model-card

Overview

Based on "Interpretable Machine Learning" by Christoph Molnar, with structure informed by the Google Model Cards framework (Mitchell et al., 2019). The core principle: a model without documentation is a liability. Interpretability is not a post-hoc courtesy - it is a prerequisite for responsible deployment. A model card makes the model's intended use, performance limits, and failure modes explicit before the model reaches production.

Workflow

Step 1: Write the model overview section

## Model Overview

**Model name:** [descriptive name, not a variable name]
**Version:** [semantic version, e.g., v1.2.0]
**Model type:** [e.g., XGBoost binary classifier, LSTM time-series forecaster]
**Owner:** [team or individual responsible for this model]
**Last updated:** [date]

**What does this model do?**
[One paragraph. Plain language. No jargon. What input does it take? What output does it produce?
What decision does it support?]

**Primary use case:**
[Specific intended application. Be concrete: "Predicts 30-day churn probability for active
subscribers with > 60 days of account history."]

**Out-of-scope uses:**
[List uses this model was NOT designed for and should not be used for.]

Step 2: Document training data and known limitations

## Training Data

**Data source:** [database, pipeline, or dataset name]
**Time range:** [e.g., Jan 2022 - Dec 2023]
**Size:** [rows and columns in the training set]
**Target variable:** [column name and class distribution, e.g., "churned: 8.3% positive"]
**Train / validation / test split:** [e.g., 70/15/15, stratified by target]

**Known data limitations:**
- [e.g., "Users from regions X and Y are underrepresented (< 2% of training data combined)"]
- [e.g., "Labels were collected through a manual review process with 15% inter-annotator disagreement"]
- [e.g., "Training data reflects behavior during a promotional period - may not generalize to steady-state"]

Step 3: Document model performance metrics

Report performance on the held-out test set. Always report more than accuracy.

## Performance

**Evaluation dataset:** [held-out test set description]

| Metric    | Value  | Notes                                |
|-----------|--------|--------------------------------------|
| AUC-ROC   | 0.847  | Primary metric                       |
| Precision | 0.71   | At threshold = 0.45                  |
| Recall    | 0.63   | At threshold = 0.45                  |
| F1        | 0.67   |                                      |
| Brier     | 0.082  | Calibration quality (lower = better) |

**Threshold selection rationale:**
[Why was this threshold chosen? What is the cost of false positives vs. false negatives?]

**Baseline comparison:**
[How does this model perform vs. the previous model or a naive baseline?]

Step 4: Document subgroup performance and fairness evaluation

Molnar's emphasis: aggregate performance metrics hide disparate performance across subgroups.

## Subgroup Performance

| Subgroup             | AUC   | Precision | Recall | Sample Size |
|----------------------|-------|-----------|--------|-------------|
| All users            | 0.847 | 0.71      | 0.63   | 50,000      |
| Mobile users         | 0.831 | 0.68      | 0.61   | 31,000      |
| Desktop users        | 0.872 | 0.76      | 0.67   | 19,000      |
| Account age < 90d    | 0.762 | 0.54      | 0.58   | 8,200       |
| Account age > 365d   | 0.889 | 0.79      | 0.71   | 22,000      |

**Known performance gaps:**
[Call out any subgroup with substantially lower performance. Document whether this is
acceptable or a known risk to remediate.]

Step 5: Document features, interpretability, and failure modes

## Features and Interpretability

**Number of input features:** [N]

**Top features by importance:**
| Feature                  | Importance | Notes                          |
|--------------------------|------------|--------------------------------|
| days_since_last_login    | 0.18       | Strongest predictor            |
| session_count_30d        | 0.14       |                                |
| support_tickets_open     | 0.11       | Proxy for dissatisfaction      |

**Interpretability method:** [SHAP, LIME, permutation importance, or none]
**Explainability at inference time:** [Yes/No - can individual predictions be explained?]

## Known Failure Modes

- **[Failure mode 1]:** [When does the model perform poorly? What data distribution causes it?]
- **[Failure mode 2]:** [e.g., "Model underperforms on accounts with < 30 days of history - cold start problem"]
- **[Failure mode 3]:** [e.g., "Performance degrades on data from promotional periods"]

Step 6: Document deployment context and monitoring plan

## Deployment

**Serving environment:** [batch job / real-time API / embedded in product]
**Inference frequency:** [daily batch / per-request / hourly]
**Latency requirement:** [e.g., < 200ms p99]
**Input data pipeline:** [where does inference data come from? what preprocessing is applied?]

## Monitoring and Maintenance

**Data drift monitoring:** [what features are monitored? what triggers a retraining alert?]
**Performance monitoring:** [what metric is tracked in production? what is the degradation threshold?]
**Retraining cadence:** [monthly / on-trigger / manual review]
**Model owner for production issues:** [team or individual]
**Deprecation plan:** [when and how will this model be retired?]

Anti-Patterns

1. Reporting only overall accuracy Bad: "The model is 91% accurate." Good: Report AUC, precision, recall at the operating threshold, and subgroup breakdowns. 91% overall can mean 60% on the segments that matter most.

2. Vague intended use statements Bad: "This model predicts user behavior." Good: "This model predicts 30-day voluntary churn for subscribers with > 60 days of account history, using account activity features from the past 90 days."

3. No out-of-scope section Bad: Documenting only what the model does, not what it should not be used for. Good: Explicitly listing misuse cases. Other teams will reuse your model for purposes you did not intend.

4. Skipping subgroup performance Bad: Publishing aggregate metrics without subgroup analysis. Good: Always stratify by the subgroups that matter for the use case. If the model supports a decision that affects people, fairness analysis is not optional.

Quality Checklist

• [ ] Model name, version, type, owner, and last-updated date present
• [ ] Intended use written in plain language with specific scope
• [ ] Out-of-scope uses listed explicitly
• [ ] Training data source, time range, size, and known limitations documented
• [ ] Test set performance reported with AUC, precision, recall, and F1 (not just accuracy)
• [ ] Subgroup performance table included for operationally relevant groups
• [ ] Top features by importance listed with interpretability method named
• [ ] At least 2 known failure modes documented
• [ ] Deployment environment and monitoring plan included