wenmin-wu/tabular-autoencoder-timeseries-embedding

Autoencoder Time-Series Embedding

Overview

When per-subject time-series data is too large or variable-length to feed directly into tabular models, compress the .describe() statistics through an autoencoder. The encoder's bottleneck produces a fixed-size dense embedding that captures non-linear relationships between summary stats. Works well when raw stats are high-dimensional (50-200 columns) and correlated.

Quick Start

import torch
import torch.nn as nn
import pandas as pd
from sklearn.preprocessing import StandardScaler

class AutoEncoder(nn.Module):
    def __init__(self, input_dim, enc_dim):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(input_dim, enc_dim * 3), nn.ReLU(),
            nn.Linear(enc_dim * 3, enc_dim * 2), nn.ReLU(),
            nn.Linear(enc_dim * 2, enc_dim), nn.ReLU())
        self.decoder = nn.Sequential(
            nn.Linear(enc_dim, input_dim * 2), nn.ReLU(),
            nn.Linear(input_dim * 2, input_dim * 3), nn.ReLU(),
            nn.Linear(input_dim * 3, input_dim), nn.Sigmoid())

    def forward(self, x):
        return self.decoder(self.encoder(x))

def encode_features(df_stats, enc_dim=50, epochs=50, bs=32):
    scaler = StandardScaler()
    X = torch.FloatTensor(scaler.fit_transform(df_stats.fillna(0)))
    ae = AutoEncoder(X.shape[1], enc_dim)
    opt = torch.optim.Adam(ae.parameters())
    for _ in range(epochs):
        for i in range(0, len(X), bs):
            batch = X[i:i+bs]
            loss = nn.MSELoss()(ae(batch), batch)
            opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        enc = ae.encoder(X).numpy()
    return pd.DataFrame(enc, columns=[f'enc_{i}' for i in range(enc_dim)])

Workflow

1. Compute df.describe() on each subject's time-series → flatten into one row per subject
2. Fill NaN with 0, then StandardScaler to normalize
3. Train autoencoder (MSE loss) on the full dataset (train + test) — unsupervised, no leakage
4. Extract encoder output as new features
5. Merge encoded features with main tabular DataFrame for GBDT training

Key Decisions

• Encoding dim: 50 is a good default for 100-200 input stats; reduce to 20-30 for smaller inputs
• Include test data: yes — autoencoder is unsupervised, using test improves the learned manifold
• Epochs: 50 is usually sufficient; monitor reconstruction loss for convergence
• Alternative: PCA/SVD is faster but misses non-linear structure

References

• LB0.494 with TabNet
• CMI | Tuning | Ensemble of solutions