wenmin-wu/tabular-lgbm-dart-boosting

LightGBM DART Boosting

Overview

DART (Dropouts meet Multiple Additive Regression Trees) randomly drops completed trees during training, reducing over-specialization of later trees. Combined with aggressive feature subsampling (20%) and bagging fraction (50%), it produces strong generalization on high-dimensional tabular data where standard GBDT overfits. Trades training speed for better holdout scores.

Quick Start

import lightgbm as lgb

params = {
    'objective': 'binary',
    'metric': 'binary_logloss',
    'boosting': 'dart',
    'num_leaves': 100,
    'learning_rate': 0.01,
    'feature_fraction': 0.20,
    'bagging_freq': 10,
    'bagging_fraction': 0.50,
    'lambda_l2': 2,
    'min_data_in_leaf': 40,
    'n_jobs': -1,
}

model = lgb.train(
    params=params,
    train_set=lgb_train,
    num_boost_round=10500,
    valid_sets=[lgb_train, lgb_valid],
    callbacks=[lgb.early_stopping(1500), lgb.log_evaluation(100)],
)

Key Decisions

• feature_fraction=0.20: aggressive subsampling forces diverse trees; increase to 0.4 for fewer features
• early_stopping=1500: DART needs a large patience since loss curves are noisier
• num_boost_round=10500: DART requires more rounds than GBDT due to dropout
• No DART at inference: all trees are used at prediction time — dropout is training-only
• Slower than GBDT: ~3-5x slower training; worth it when overfitting gap is large

References

• Source: amex-lgbm-dart-cv-0-7977
• Competition: American Express - Default Prediction