wenmin-wu/tabular-tfidf-weighted-category-counts

Overview

Categorical count features (how often each activity / key / event type occurred per session) suffer from two problems: long sessions dominate raw counts, and common categories drown out rare-but-informative ones. Applying TF-IDF — the same transform used on text term counts — fixes both. Normalize each count by the session total (TF) and weight by how rare the category is across sessions (IDF). The result behaves like a dense numeric feature but preserves rarity signal. Fit IDF on train only, then reuse on test to avoid leakage.

Quick Start

import numpy as np

class CategoryTfidf:
    def __init__(self):
        self.idf = {}

    def fit_transform(self, counts_df):
        # counts_df: rows = session id, columns = categories, values = counts
        cnts = counts_df.sum(axis=1)
        out = counts_df.copy().astype(float)
        N = len(counts_df)
        for col in counts_df.columns:
            df_col = (counts_df[col] > 0).sum()
            idf = np.log(N / (df_col + 1))
            self.idf[col] = idf
            tf = counts_df[col] / cnts.replace(0, 1)
            out[col] = (1 + np.log1p(tf)) * idf
        return out

    def transform(self, counts_df):
        cnts = counts_df.sum(axis=1)
        out = counts_df.copy().astype(float)
        for col in counts_df.columns:
            tf = counts_df[col] / cnts.replace(0, 1)
            out[col] = (1 + np.log1p(tf)) * self.idf.get(col, 0)
        return out

Workflow

1. Build a (session × category) count matrix — pivot with crosstab or groupby().size().unstack()
2. Fit TF-IDF on the training split: store per-category IDF
3. Transform both train and test using the stored IDF values
4. Concatenate with other tabular features and feed to LGBM / NN
5. Validate: on held-out OOF, TF-IDF typically beats raw counts by 0.3-1% in RMSE / AUC for long-tail categoricals

Key Decisions

• Fit IDF on train only: fitting on train+test leaks the category distribution of test. Always split first.
• +1 smoothing: log(N/(df+1)) avoids division-by-zero when a category appears only in test.
• vs. raw counts: raw counts are fine when the category vocabulary is small (<10); TF-IDF wins when >30 categories with long tail.
• vs. sklearn TfidfVectorizer: Vectorizer expects token lists; this applies the same math to pre-counted data directly.

References

• Silver Bullet | Single Model | 165 Features