brycewang-stanford/robustness

Robustness Test Checklist

Systematic checklist of robustness tests for empirical research. Use this to ensure comprehensive testing before submission.

---

Core Robustness Categories

1. Alternative Samples

| Test | Description | When to Use | |------|-------------|-------------| | Exclude outliers | Winsorize/trim at different levels (0.5%, 2%, 5%) | Always | | Drop financial firms | Exclude SIC 6000-6999 | If not already excluded | | Drop regulated industries | Exclude utilities, telecoms | Industry-specific effects | | Different time periods | Split sample pre/post crisis, early/late | Results may be period-specific | | Geographic subsamples | By region, state, country | External validity | | Size subsamples | Small vs. large firms | Heterogeneous effects | | Balanced panel | Require continuous observations | Survivorship concerns |

2. Alternative Specifications

| Test | Description | When to Use | |------|-------------|-------------| | Different fixed effects | Firm, industry×year, state×year | Control for unobservables | | Additional controls | Add variables referees might suggest | Omitted variable concerns | | Drop controls | Verify not over-controlling | Mediator concerns | | Different clustering | Firm, industry, state, two-way | Inference robustness | | Different standard errors | Bootstrap, Newey-West, Driscoll-Kraay | Serial/cross-sectional correlation | | Nonlinear specifications | Quadratic terms, splines | Linearity assumption | | Log vs. level | Transform dependent variable | Skewed distributions |

3. Alternative Measures

| Test | Description | When to Use | |------|-------------|-------------| | Alternative dependent variable | Different proxy for same concept | Measurement concerns | | Alternative treatment measure | Continuous vs. binary, different threshold | Treatment definition | | Alternative control measures | Different proxies for size, leverage, etc. | Standard practice | | Scaled differently | By assets, sales, employees | Scaling choice matters |

4. Identification Tests

| Test | Description | When to Use | |------|-------------|-------------| | Placebo/Falsification | | | | Placebo timing | Fake treatment 1-3 years before actual | DiD parallel trends | | Placebo outcome | Effect on outcome that shouldn't be affected | Specificity of mechanism | | Placebo treatment | Random assignment of treatment | Rule out spurious correlation | | Pre-trends | | | | Event study plot | Coefficient for each pre/post period | Visual parallel trends | | Joint F-test | Test pre-period coefficients = 0 | Statistical parallel trends | | Endogeneity | | | | Instrumental variables | Find exogenous variation | Selection concerns | | Heckman selection | Model selection explicitly | Sample selection | | Propensity score matching | Match treated/control | Observable selection | | Entropy balancing | Reweight to balance covariates | Covariate imbalance | | Regression discontinuity | If threshold exists | Sharp identification |

5. Inference Robustness

| Test | Description | When to Use | |------|-------------|-------------| | Wild cluster bootstrap | Small number of clusters | <50 clusters | | Randomization inference | Permutation-based p-values | Few treated units | | Conley standard errors | Spatial correlation | Geographic data | | Multiple hypothesis correction | Bonferroni, FDR | Many outcomes tested |

---

DiD-Specific Tests

For difference-in-differences designs:

• [ ] Event study with pre-treatment coefficients
• [ ] Parallel trends test (formal)
• [ ] Bacon decomposition (staggered treatment)
• [ ] Callaway-Sant'Anna or Sun-Abraham estimator (heterogeneous treatment effects)
• [ ] Placebo treatment timing
• [ ] Vary treatment window
• [ ] Triple-difference if possible
• [ ] Exclude always-treated or never-treated

---

IV-Specific Tests

For instrumental variables:

• [ ] First-stage F-statistic (>10, prefer >100)
• [ ] Weak instrument robust inference (Anderson-Rubin)
• [ ] Overidentification test (if multiple IVs)
• [ ] Exclusion restriction discussion
• [ ] Reduced form results
• [ ] Compare OLS vs. IV magnitudes

---

Quick Commands

• "robustness checklist" - Full checklist for current paper
• "DiD robustness" - DiD-specific tests only
• "what tests for [method]" - Tests for specific identification strategy
• "referee-proof" - Most commonly requested tests
• "prioritize tests" - Rank by importance for your setting

---

Referee-Proof Minimum

At minimum, most papers should include:

1. Sample robustness: Exclude outliers, alternative time periods
2. Specification robustness: Alternative fixed effects, controls
3. Measurement robustness: Alternative variable definitions
4. Inference robustness: Alternative clustering
5. Identification test: At least one placebo or pre-trend test

---

Reporting Template

For robustness tables:

Table X: Robustness Tests
Panel A: Alternative Samples
  (1) Baseline
  (2) Exclude financial firms
  (3) Exclude 2008-2009
  (4) Winsorize at 5%

Panel B: Alternative Specifications
  (5) Add industry×year FE
  (6) Control for firm age
  (7) Cluster by industry

Panel C: Alternative Measures
  (8) Alternative dependent variable
  (9) Continuous treatment measure