wenmin-wu/timeseries-store-profile-hierarchical-clustering

Overview

Vendor-supplied store type and cluster labels (common in retail datasets like Favorita) usually encode store size, not demand shape. A store profile is the right grouping key when you plan to train per-group models or apply per-group post-processing: two small stores with identical weekday shapes should cluster together even if one sells twice as much. The recipe is three steps — normalize each row by its store's overall mean to kill scale, aggregate to a per-store weekday mean+std profile, then Ward-cluster the profiles.

Quick Start

import pandas as pd
from scipy.cluster.hierarchy import ward, dendrogram
from sklearn.cluster import AgglomerativeClustering

store_avg = sales.groupby('store_nbr')['transactions'].mean()
sales['normalized'] = sales['transactions'] / sales['store_nbr'].map(store_avg)

profile = (sales.groupby(['store_nbr', 'day_of_week'])['normalized']
                .agg(['mean', 'std'])
                .unstack(level='day_of_week'))   # (n_stores, 14)

dendrogram(ward(profile.values))                 # eyeball a cutoff k
labels = AgglomerativeClustering(n_clusters=6).fit_predict(profile.values)
stores['shape_cluster'] = labels

Workflow

1. Compute each store's overall mean target and divide every row by it → scale-invariant normalized target
2. Group by (store, day_of_week) and aggregate mean and std → unstack so each store is one feature row (7 × 2 = 14 cols)
3. Optionally add a second unstack by a dayoff flag for 28-dim profiles that separate holiday behavior
4. Plot a Ward dendrogram on the profile matrix and pick k at the biggest vertical gap
5. Fit AgglomerativeClustering(k) on the profile matrix and attach labels to the stores table
6. Use shape_cluster as a grouping key for per-cluster models or as a categorical feature in a global model

Key Decisions

• Normalize before aggregating: raw means are dominated by store size and destroy shape clustering.
• Mean + std, not just mean: two stores with equal weekday averages but different volatility behave differently on promo days.
• Ward linkage: produces compact equal-variance clusters well-matched to mean/std features; single-linkage produces chains.
• Pick k from dendrogram gap: silhouette is unreliable on 14-dim profiles; visual inspection is faster and more interpretable.
• Discard vendor labels: if the dendrogram shows clear shape groups that don't match the vendor's type/cluster, the vendor labels are noise.

References

• A first Kaggle - Part 1 - Forecasting store #47