wenmin-wu/cv-per-label-platt-isotonic-calibration

Overview

Multi-label deep nets are systematically miscalibrated: rare labels are pushed to extreme low probabilities, common labels saturate at high ones. Calibration fixes this without retraining. The recipe: collect out-of-fold scores per label across the training set, fit a LogisticRegression per label (Platt scaling), and gracefully fall back to IsotonicRegression for labels where Platt fails (constant scores, single-class folds, divergence). Save the dict {label: ('platt'|'isotonic', model)} to disk and apply at inference. The lift is usually 0.001-0.003 on macro metrics but it's free, deterministic, and stacks with all other tricks.

Quick Start

import numpy as np, joblib
from sklearn.linear_model import LogisticRegression
from sklearn.isotonic import IsotonicRegression

def fit_calibrators(oof_df, gt_df, label_cols):
    cal = {}
    for col in label_cols:
        s = oof_df[col].values
        y = gt_df[col].values
        if np.unique(y).size < 2 or np.allclose(s, s[0]):
            cal[col] = None
            continue
        try:
            lr = LogisticRegression(max_iter=2000)
            lr.fit(s.reshape(-1, 1), y)
            cal[col] = ('platt', lr)
        except Exception:
            iso = IsotonicRegression(out_of_bounds='clip')
            iso.fit(s, y)
            cal[col] = ('isotonic', iso)
    return cal

def apply_calibrators(scores, cal, label_cols):
    out = scores.copy()
    for i, col in enumerate(label_cols):
        c = cal.get(col)
        if c is None: continue
        kind, m = c
        if kind == 'platt':
            out[:, i] = m.predict_proba(scores[:, i].reshape(-1, 1))[:, 1]
        else:
            out[:, i] = m.transform(scores[:, i])
    return out

Workflow

1. Generate out-of-fold predictions from your CV pipeline — never use train predictions
2. For each label column, fit Platt scaling; fall back to isotonic on failure
3. Skip labels with constant scores or single-class targets
4. Pickle the calibrator dict alongside the model checkpoint
5. At inference, transform raw sigmoid scores through the corresponding calibrator
6. Re-tune any per-label thresholds after calibration — they shift slightly

Key Decisions

• Per-label calibrators, not global: each label has its own miscalibration curve.
• Platt before isotonic: Platt is parametric (1 param), generalizes better with little data; isotonic needs more samples to be stable.
• Skip degenerate labels gracefully: a label with no positives in OOF will crash both fitters — return None and pass scores through unchanged.
• Calibrate before threshold tuning: doing it after invalidates the threshold choices.
• Out-of-fold is mandatory: in-fold calibration is overconfident and inflates validation.
• Lift is small but free: don't expect leaderboard miracles, but at zero training cost, always worth turning on for the final submission.

References

• RSNA Notebook