wenmin-wu/tabular-optuna-lgbm-tuning

Optuna LightGBM Tuning

Overview

Optuna's Tree-structured Parzen Estimator (TPE) efficiently searches LightGBM hyperparameter space by building a probabilistic model of which regions produce good scores. Typically finds better parameters than grid/random search in fewer trials. Define an objective function, let Optuna propose params, evaluate with CV.

Quick Start

import optuna
import lightgbm as lgb
from sklearn.model_selection import cross_val_score

def objective(trial):
    params = {
        'n_estimators': trial.suggest_int('n_estimators', 100, 2000),
        'num_leaves': trial.suggest_int('num_leaves', 16, 256),
        'max_depth': trial.suggest_int('max_depth', 3, 12),
        'learning_rate': trial.suggest_float('learning_rate', 1e-3, 0.3, log=True),
        'min_child_samples': trial.suggest_int('min_child_samples', 5, 100),
        'subsample': trial.suggest_float('subsample', 0.5, 1.0),
        'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),
        'reg_alpha': trial.suggest_float('reg_alpha', 1e-8, 10.0, log=True),
        'reg_lambda': trial.suggest_float('reg_lambda', 1e-8, 10.0, log=True),
    }
    model = lgb.LGBMClassifier(**params, random_state=42, n_jobs=-1)
    score = cross_val_score(model, X_train, y_train, cv=5, scoring='roc_auc').mean()
    return score

study = optuna.create_study(direction='maximize', sampler=optuna.samplers.TPESampler(seed=42))
study.optimize(objective, n_trials=100, show_progress_bar=True)

best_model = lgb.LGBMClassifier(**study.best_params)
best_model.fit(X_train, y_train)

Workflow

1. Define objective function with trial.suggest_* for each hyperparameter
2. Use log scale for learning rate and regularization params
3. Run 50-200 trials (TPE converges fast)
4. Extract best params and retrain on full data
5. Optionally use optuna.visualization to inspect param importance

Key Decisions

• TPE vs random: TPE finds good regions 2-3x faster than random search
• n_trials: 50 for quick search, 200 for thorough; diminishing returns after ~100
• Pruning: Add optuna.integration.LightGBMPruningCallback to early-stop bad trials
• Seed: Fix sampler seed for reproducibility across runs

References

• Riiid Answer Correctness Prediction (Kaggle)
• Source: riiid-answer-correctness-prediction-eda-modeling