wenmin-wu/timeseries-retroactive-outlier-rescaling

Retroactive Outlier Rescaling

Overview

Administrative time series (business counts, registrations, economic indicators) sometimes contain abrupt level shifts caused by data collection changes rather than real trends. Walking backward from the most recent value and rescaling earlier segments when the jump exceeds a threshold (e.g., 20% of running mean) corrects these artifacts while preserving genuine trends.

Quick Start

import numpy as np

def fix_outliers(series, threshold_frac=0.20):
    values = series.values.copy().astype(float)
    n = len(values)
    for i in range(n - 2, -1, -1):
        running_mean = np.mean(values[i + 1 : min(i + 4, n)])
        if running_mean == 0:
            continue
        jump = abs(values[i] - values[i + 1])
        if jump > threshold_frac * running_mean:
            scale = values[i + 1] / values[i] if values[i] != 0 else 1.0
            values[: i + 1] *= scale
    return values

df["value_clean"] = df.groupby("entity_id")["value"].transform(fix_outliers)

Workflow

1. Group by entity, sort by time (most recent last)
2. Walk backward from the second-to-last observation
3. At each step, compute the running mean of the next 2-3 values
4. If |value[i] - value[i+1]| exceeds threshold_frac * running_mean, compute scale factor
5. Multiply all values before index i by value[i+1] / value[i]
6. Use cleaned series for both training and inference

Key Decisions

• Threshold: 0.20 (20%) catches major level shifts without overcorrecting normal volatility
• Direction: backward walk anchors to the most recent (most reliable) data
• Running window: 2-3 points balances stability vs. sensitivity to recent changes
• Multiplicative vs. additive: multiplicative preserves proportional relationships across entities with different scales

References

• Better XGB Baseline
• GoDaddy: Tune Stacking