a-green-hand-jack/statistical-analysis-planner

Statistical Analysis Planner

Design the statistical analysis before running, and report it correctly after results exist. This skill prevents underpowered claims, misleading averages-without-variance, and significance theater in ML papers.

Use this skill when:

• deciding which significance tests or confidence intervals to report for a result table
• seed variance is high and a single-run result may not be representative
• comparing methods and wanting to know if the difference is statistically meaningful
• a paper or rebuttal needs to defend a claim quantitatively against reviewer variance concerns
• an ablation result is close and the decision to include it depends on whether the difference is real
• multiple comparisons are being made and type-I error accumulation needs to be controlled

Do not use this skill to run the experiments — use run-experiment. Do not use this skill to interpret surprising results scientifically — use result-diagnosis. Use this skill after results exist (or in planning mode before deciding how many seeds to run).

Pair this skill with:

• experiment-design-planner to plan the number of seeds, runs, and controls before running
• result-diagnosis when the statistical analysis reveals that a result is not reliable
• paper-evidence-board to update evidence slots with confidence-annotated claims
• table-results-review to ensure result tables report variance and pass statistical requirements

Skill Directory Layout

<installed-skill-dir>/
├── SKILL.md
└── references/
    └── test-selection.md

Progressive Loading

• Always read references/test-selection.md when choosing a statistical test or confidence interval method.
• Read memory/claim-board.md and memory/evidence-board.md to understand what claims need statistical backing.

Core Principles

• A mean result without variance is not an empirical claim — it is an anecdote.
• Report the number of seeds and independent runs, not just the metric value.
• Choose the test before seeing the results, not after. Post-hoc test selection biases results.
• Effect size matters more than p-value for practical significance in ML.
• Multiple comparisons require corrections. If you test 10 ablations, 0.5 of them will be "significant" at p<0.05 by chance.
• Reviewer variance concerns are common at NeurIPS/ICLR. Anticipate them with pre-planned variance analysis.
• If compute prevents many seeds, acknowledge the limitation explicitly rather than overclaiming.

Step 1 — Identify What Needs Statistical Analysis

For each result that will appear in the paper, record:

• the claim being made ("Method A outperforms Baseline B on Task C")
• the metric and its expected distribution
• how many independent runs (seeds) exist
• whether the comparison is within-subject (same data, different methods) or between-subject (different data splits)

Classify each result as:

• requires-analysis: main claim or primary comparison
• supporting-analysis: ablation or secondary result
• descriptive-only: mean reported, no significance claim
• single-run: only one run exists, limitations must be acknowledged

Step 2 — Choose the Analysis Plan

Read references/test-selection.md.

For each requires-analysis result:

Result: <claim or comparison>
Metric: <metric name>
N seeds / runs: <count>
Distribution assumption: normal / non-normal / unknown
Test: <paired t-test / Wilcoxon / bootstrap CI / permutation test / McNemar>
Significance threshold: α = 0.05 (or 0.01 for primary claim)
Effect size measure: Cohen's d / Cliff's delta / relative improvement %
Multiple comparison correction: <Bonferroni / Holm / Benjamini-Hochberg / none>
Report format: mean ± std / 95% CI / p-value + effect size

For seed variance analysis, plan:

• minimum number of seeds to detect the expected effect size at power 0.8
• how to report variance: standard deviation across seeds, bootstrap CI, or min/max range

Step 3 — Run or Verify the Analysis

For results that already exist, compute:

• mean and standard deviation across seeds
• 95% confidence interval (bootstrap recommended for non-normal distributions)
• p-value from the chosen test (if significance is being claimed)
• effect size (Cohen's d or relative improvement %)
• corrected p-values if multiple comparisons are made

For compute-limited settings (1–3 seeds):

• report mean and range (min/max) rather than standard deviation
• acknowledge the limitation explicitly in the paper
• do not claim statistical significance with fewer than 5 independent runs for parametric tests

Step 4 — Report Format for Paper

For main result tables:

Method A: 82.3 ± 1.2 (mean ± std, N=5 seeds)
          [80.4, 84.1] 95% CI
          p < 0.05 vs Baseline B (paired t-test, Bonferroni-corrected)
          Effect size: d = 0.83 (large)

For text claims:

• "X outperforms Y by Z% (p < 0.05, d = 0.6)" is preferred over "X significantly outperforms Y"
• "X achieves [metric] = A ± B across N seeds" is preferred over "X achieves A"
• Avoid "significantly" without a reported test and threshold

For low-seed settings:

• "X achieves [metric] = A (range: [B, C], N=3 seeds); we note this result is based on limited seeds"

Step 5 — Multiple Comparison Audit

If the paper reports more than 3 comparisons on the same held-out set:

• list all comparisons
• apply Bonferroni correction (divide α by number of tests) or Holm correction (less conservative)
• flag any comparison that loses significance after correction
• decide whether to include or describe as "trend" rather than "significant"

Memory Writeback

• Update memory/evidence-board.md when statistical analysis changes the confidence level of a claim
• Update memory/claim-board.md to reflect corrected or strengthened claim wording
• Update memory/risk-board.md when low seed count or failed significance is a reviewer risk

Final Sanity Check

Before finalizing:

• every main result table row has at least N, mean, and variance reported
• significance tests were chosen before seeing the specific results, or the analysis plan was declared a priori
• multiple-comparison corrections are applied when > 3 comparisons share a test set
• effect sizes are reported alongside p-values for claimed differences
• compute-limited seed counts are acknowledged as limitations
• claims in the paper match the statistical evidence (no overclaiming)