wenmin-wu/cv-multilabel-auc-evaluation

Multi-Label AUC Evaluation

Overview

For multi-label classification (multiple binary labels per sample), compute ROC-AUC independently for each label, then average. This macro-averaged AUC treats rare and common labels equally, preventing the model from ignoring minority classes. Use as the primary metric for model selection and early stopping.

Quick Start

import numpy as np
from sklearn.metrics import roc_auc_score

def multilabel_auc(y_true, y_pred, label_names=None):
    """Compute per-label and macro-averaged AUC.

    Args:
        y_true: array (n_samples, n_labels), binary ground truth
        y_pred: array (n_samples, n_labels), predicted probabilities
        label_names: optional list of label names for reporting
    """
    per_label = {}
    for i in range(y_true.shape[1]):
        # Skip labels with single class in this split
        if len(np.unique(y_true[:, i])) < 2:
            continue
        name = label_names[i] if label_names else f'label_{i}'
        per_label[name] = roc_auc_score(y_true[:, i], y_pred[:, i])

    macro_auc = np.mean(list(per_label.values()))
    return macro_auc, per_label

# Usage in validation loop
y_preds = model(images).sigmoid().cpu().numpy()
macro_auc, per_label = multilabel_auc(y_true, y_preds, target_cols)
print(f"Macro AUC: {macro_auc:.4f}")
for name, auc in per_label.items():
    print(f"  {name}: {auc:.4f}")

Workflow

1. Collect all validation predictions (sigmoid probabilities, not logits)
2. Compute ROC-AUC per label independently
3. Skip labels with only one class in the validation set (AUC undefined)
4. Macro-average across all valid labels
5. Use macro AUC for early stopping and model checkpointing

Key Decisions

• Macro vs micro: Macro weights all labels equally; micro favors majority labels
• Sigmoid vs softmax: Multi-label uses sigmoid (independent per label); multi-class uses softmax
• Threshold-free: AUC doesn't require choosing a threshold — better for model selection
• Per-label tracking: Monitor per-label AUC to catch labels where the model struggles

References

• RANZCR CLiP - Catheter and Line Position Challenge (Kaggle)
• Source: ranzcr-resnext50-32x4d-starter-training