wenmin-wu/tabular-synthetic-sample-detection

Synthetic Sample Detection

Overview

Some competitions inject synthetic rows into the test set to prevent probing or inflate leaderboard noise. A reliable signal: real data points almost always have at least one feature value that appears only once across the entire dataset, while synthetic rows (generated by sampling from existing value distributions) lack any truly unique values. Flag rows with zero unique values as synthetic and exclude them from frequency-based feature engineering.

Quick Start

import numpy as np

df_test = test_df.drop("ID_code", axis=1).values

unique_count = np.zeros_like(df_test)
for col in range(df_test.shape[1]):
    _, idx, counts = np.unique(df_test[:, col], return_index=True, return_counts=True)
    unique_count[idx[counts == 1], col] += 1

has_unique = np.sum(unique_count, axis=1) > 0
real_idx = np.argwhere(has_unique)[:, 0]
fake_idx = np.argwhere(~has_unique)[:, 0]

print(f"Real: {len(real_idx)}, Synthetic: {len(fake_idx)}")

Workflow

1. For each feature column, find values that appear exactly once
2. Mark which rows contain those unique values
3. Sum unique-value flags per row: rows with sum > 0 are real
4. Rows with zero unique values are synthetic
5. Use only real test rows when computing frequency-based features

Key Decisions

• Threshold: Sum > 0 is the standard cutoff; more lenient thresholds catch edge cases
• Use case: Frequency encoding, count-based features should exclude synthetic rows
• Prediction: Still predict on all test rows (real + synthetic) for submission
• Generality: Works when synthetics are generated by independent column sampling

References

• List of Fake Samples and Public/Private LB split
• 200 Magical Models - Santander - [0.920]