wenmin-wu/tabular-threaded-parquet-describe-features

Threaded Parquet Describe Features

Overview

When time-series data is stored as one parquet file per subject (common in wearable/sensor competitions), sequential loading is slow. Use ThreadPoolExecutor to read all files in parallel, compute df.describe() per subject, and flatten the 8-stat-per-column summary into a single feature row. Converts variable-length time-series into fixed-width tabular features in seconds.

Quick Start

import os
import pandas as pd
from concurrent.futures import ThreadPoolExecutor

def process_subject(subject_dir, base_dir):
    path = os.path.join(base_dir, subject_dir, 'part-0.parquet')
    df = pd.read_parquet(path)
    df = df.drop(columns=['step'], errors='ignore')
    stats = df.describe().values.reshape(-1)
    sid = subject_dir.split('=')[-1]
    return stats, sid

def load_timeseries_features(base_dir):
    subjects = os.listdir(base_dir)
    with ThreadPoolExecutor() as pool:
        results = list(pool.map(
            lambda s: process_subject(s, base_dir), subjects))
    stats, ids = zip(*results)
    n_feats = len(stats[0])
    df = pd.DataFrame(list(stats),
                       columns=[f'ts_stat_{i}' for i in range(n_feats)])
    df['id'] = ids
    return df

ts_features = load_timeseries_features('data/series_train.parquet/')
train = train.merge(ts_features, on='id', how='left')

Workflow

1. List all subject directories under the parquet base path
2. ThreadPoolExecutor.map() reads each parquet and computes describe()
3. Flatten the 8-row x N-column describe output into a 1D vector per subject
4. Assemble into a DataFrame and merge with main table on subject ID

Key Decisions

• ThreadPoolExecutor over ProcessPool: parquet reads are I/O-bound, thread pool avoids pickling overhead
• describe() gives 8 stats: count, mean, std, min, 25%, 50%, 75%, max — good baseline coverage
• Drop step column: monotonic index adds no information
• Naming: generic ts_stat_i columns; map back via (stat_idx, col_name) if interpretability needed

References

• CMI | Best Single Model