wenmin-wu/tabular-sparse-dense-hstack-lgbm

Overview

When you have both wide sparse text features (TF-IDF, CountVectorizer — 50k to 500k columns) and dense tabular columns, the wrong move is to densify the sparse block (OOM) or to SVD-compress it (loses signal). LightGBM accepts CSR matrices natively, so you can scipy.sparse.hstack the text block and the dense predictors into one CSR matrix and hand it straight to lgb.Dataset. The trick is preserving feature names and the categorical-feature flag across the boundary, so LightGBM still uses its native categorical splits on the label-encoded columns living inside the otherwise-sparse matrix.

Quick Start

import scipy.sparse as sp
import numpy as np
import lightgbm as lgb

X_train = sp.hstack([
    train_desc_tfidf,       # (n, ~100k) sparse
    train_title_counts,     # (n, ~20k) sparse
    train[dense_cols].values,  # (n, d) dense
], format='csr')

feature_names = np.hstack([
    tfidf.get_feature_names_out(),
    cv.get_feature_names_out(),
    dense_cols,
]).tolist()

dtrain = lgb.Dataset(
    X_train, label=y_train,
    feature_name=feature_names,
    categorical_feature=categorical_cols,
)

params = dict(objective='regression', metric='rmse',
              num_leaves=32, max_depth=15, learning_rate=0.02,
              feature_fraction=0.6, bagging_fraction=0.8, bagging_freq=5)

model = lgb.train(params, dtrain, valid_sets=[dtrain, dvalid],
                  num_boost_round=16000, early_stopping_rounds=500)

Workflow

1. Fit each text vectorizer on train + test concatenated to share vocabulary
2. Label-encode categorical columns with fit on the train+test union (avoids unseen labels)
3. sp.hstack([sparse_blocks..., dense.values], format='csr') — CSR is required by LightGBM
4. Build the aligned feature_name list by concatenating get_feature_names_out() from each block and the dense column names
5. Pass both feature_name and categorical_feature (as a list of names) to lgb.Dataset so native categorical splits still work
6. Train with a low feature_fraction (0.3-0.6) to subsample the wide feature space each tree

Key Decisions

• CSR, not CSC or COO: LightGBM converts internally but errors on other layouts. format='csr' is mandatory.
• feature_fraction ≤ 0.6: with 100k+ columns, unrestricted trees slow to a crawl and overfit — subsample every split.
• Pass categoricals by name, not index: indices change if the block widths change; names are stable.
• Fit encoders/vectorizers on train+test union: unseen categorical labels or TF-IDF tokens at inference break the column alignment.
• vs. SVD-compressing the text: SVD loses rare-token signal. Sparse hstack keeps everything and lets the tree choose.
• Clip predictions to target range: np.clip(preds, 0, 1) for bounded targets like deal probability — LightGBM regression is unconstrained.

References

• Aggregated features & LightGBM