wenmin-wu/tabular-ngram-composite-features

N-gram Composite Features

Overview

Tree-based models can learn feature interactions, but explicit composite categoricals often improve performance by giving the model pre-built interaction signals. This technique selects the top categorical columns (by feature importance), creates all 2-way and 3-way string concatenations, then target-encodes the resulting high-cardinality composites.

Quick Start

from itertools import combinations
import pandas as pd

# 1. Define top categoricals (from feature importance)
TOP_CATS = [
    'Contract', 'InternetService', 'PaymentMethod',
    'OnlineSecurity', 'TechSupport', 'PaperlessBilling'
]

# 2. Create bi-gram composites
for c1, c2 in combinations(TOP_CATS, 2):
    df[f"BG_{c1}_{c2}"] = df[c1].astype(str) + "_" + df[c2].astype(str)

# 3. Create tri-gram composites (optional, use fewer columns)
for c1, c2, c3 in combinations(TOP_CATS[:4], 3):
    df[f"TG_{c1}_{c2}_{c3}"] = (
        df[c1].astype(str) + "_" + df[c2].astype(str) + "_" + df[c3].astype(str)
    )

# 4. Target-encode composites (use KFold to avoid leakage)
from sklearn.model_selection import KFold

def target_encode_kfold(df, col, target, n_splits=5):
    encoded = pd.Series(index=df.index, dtype=float)
    kf = KFold(n_splits=n_splits, shuffle=True, random_state=42)
    for tr_idx, val_idx in kf.split(df):
        means = df.iloc[tr_idx].groupby(col)[target].mean()
        encoded.iloc[val_idx] = df.iloc[val_idx][col].map(means)
    return encoded.fillna(df[target].mean())

composite_cols = [c for c in df.columns if c.startswith(("BG_", "TG_"))]
for col in composite_cols:
    df[f"TE_{col}"] = target_encode_kfold(df, col, 'target')

Workflow

1. Identify top categoricals from a baseline model's feature importance.
2. Generate bi-grams (all 2-combinations of top N columns). Use N <= 8 to keep count manageable.
3. Optionally generate tri-grams from top 4-5 columns only (combinatorics explode quickly).
4. Target-encode each composite using KFold to prevent leakage.
5. Train model with original features + target-encoded composites.
6. Prune composites with near-zero importance after first training round.

Key Decisions

| Decision | Guidance | |---|---| | How many base columns | 5-8 for bi-grams, 3-5 for tri-grams. C(8,2)=28 bi-grams is fine; C(8,3)=56 tri-grams gets heavy. | | Encoding method | KFold target encoding is standard. Alternatives: CatBoost native encoding, leave-one-out. | | Leakage prevention | Never encode using the full dataset. Always use KFold or fit on train / transform on test. | | Rare categories | Composites create many rare groups. Apply smoothing or min-count thresholds during encoding. | | When to use | Most effective when categoricals have strong interactions and the dataset has >10k rows. |

References

• Kaggle: "S6E3 Ridge XGB N-gram 0.91927 CV" (playground-series-s6e3)
• Micci-Barreca, D. "A preprocessing scheme for high-cardinality categorical attributes in classification and prediction problems" (SIGKDD Explorations, 2001)