wenmin-wu/tabular-logit-transform-stacking

Logit Transform Stacking

Overview

When stacking binary classification models, the base model outputs are probabilities in [0, 1]. Fitting a linear meta-learner directly on probabilities is suboptimal because the relationship between probabilities and log-odds is nonlinear — a 0.01 difference near 0.5 means less than near 0.99. Applying logit (log(p/(1-p))) transforms probabilities to log-odds space where linear combination is theoretically justified. The logistic regression meta-learner then learns optimal weights in this space, often outperforming probability-space blending by 0.001–0.003 AUC.

Quick Start

import numpy as np
import pandas as pd
from scipy.special import logit, expit
from sklearn.linear_model import LogisticRegression

ALMOST_ZERO = 1e-10
ALMOST_ONE = 1 - ALMOST_ZERO

# Load OOF predictions from base models
oof_preds = {}
for name, path in base_model_files.items():
    probs = pd.read_csv(path)['prediction']
    oof_preds[name] = logit(probs.clip(ALMOST_ZERO, ALMOST_ONE))

X_train = pd.DataFrame(oof_preds).values
y_train = true_labels

# Fit meta-learner in logit space
meta = LogisticRegression(C=1.0)
meta.fit(X_train, y_train)

# Inspect learned weights
weights = meta.coef_[0] / meta.coef_[0].sum()
print("Model weights:", dict(zip(oof_preds.keys(), weights)))

# Apply to test predictions
test_preds = {}
for name, path in test_model_files.items():
    probs = pd.read_csv(path)['prediction']
    test_preds[name] = logit(probs.clip(ALMOST_ZERO, ALMOST_ONE))

X_test = pd.DataFrame(test_preds).values
final_probs = meta.predict_proba(X_test)[:, 1]

Workflow

1. Generate OOF predictions from each base model
2. Clip probabilities to [1e-10, 1-1e-10] to avoid infinite logit values
3. Apply logit() transform to all base model outputs
4. Fit LogisticRegression on logit-transformed OOF predictions
5. Transform test predictions the same way, then predict with meta-learner

Key Decisions

• Clipping: Essential — logit(0) = -inf, logit(1) = inf. Clip to [1e-10, 1-1e-10]
• Regularization: C=1.0 default; lower C for many correlated base models
• vs probability blending: Logit stacking is strictly better when base models vary in calibration
• Weight interpretation: Normalized coefficients ≈ blending weights, but in log-odds space

References

• Simple Linear Stacking (LB .9730)