wenmin-wu/tabular-regularized-qda-classifier

Regularized QDA Classifier

Overview

Quadratic Discriminant Analysis models each class as a multivariate Gaussian with its own covariance matrix. When features are truly Gaussian-distributed (synthetic data, physical measurements), QDA can outperform tree-based models by directly modeling the decision boundary. The reg_param shrinks per-class covariance toward the pooled estimate, preventing singularity on high-dimensional or small-sample data.

Quick Start

from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import roc_auc_score
import numpy as np

oof = np.zeros(len(X))
preds = np.zeros(len(X_test))

skf = StratifiedKFold(n_splits=11, shuffle=True, random_state=42)
for train_idx, val_idx in skf.split(X, y):
    clf = QuadraticDiscriminantAnalysis(reg_param=0.5)
    clf.fit(X[train_idx], y[train_idx])
    oof[val_idx] = clf.predict_proba(X[val_idx])[:, 1]
    preds += clf.predict_proba(X_test)[:, 1] / skf.n_splits

print(f'AUC: {roc_auc_score(y, oof):.4f}')

Workflow

1. Identify that data has Gaussian structure (scatter plots show elliptical clusters)
2. Apply feature selection (VarianceThreshold) to remove noise features
3. Standardize features (optional — QDA is scale-invariant but helps numerically)
4. Train QDA with reg_param in [0.1, 0.5] via cross-validation
5. Average fold predictions for test set

Key Decisions

• reg_param: 0.0 = full QDA (each class has own covariance); 1.0 = LDA (shared covariance). Start at 0.5
• When to use: data generated by make_classification, physical sensor data, or any setting where classes form elliptical clusters
• Feature count: QDA estimates O(p^2) parameters per class — reduce features first if p > 50
• vs LDA: QDA wins when classes have different covariance shapes; LDA wins when covariances are similar

References

• Pseudo Labeling - QDA - [0.969]
• Instant Gratification (QDA)