qa-aman/experiment-design

Overview

Based on "The Art of Statistics" by David Spiegelhalter. The core principle: statistical significance is not the same as practical significance, and a poorly designed experiment produces confidently wrong answers. Rigorous experiment design means defining the question, estimating the required sample size, and setting success criteria before any data is collected - then interpreting results within the limits of what the experiment can actually prove.

Workflow

Step 1: State the hypothesis in falsifiable form

A hypothesis must specify: the intervention, the population, the outcome metric, and the expected direction.

Format: "Applying [intervention] to [population] will [increase/decrease] [metric] compared to the control condition."

Example: "Showing personalized recommendations to new users in the first session will increase 7-day retention compared to showing trending content."

Also state the null hypothesis explicitly: "There is no difference in 7-day retention between personalized and trending content groups."

Spiegelhalter's warning: if you cannot state what result would cause you to reject your hypothesis, you do not have a hypothesis - you have a belief.

Step 2: Identify the primary metric and guard metrics

Primary metric: The single number that determines whether the experiment succeeded. Choose one. Multiple primary metrics make interpretation ambiguous.

Guard metrics: Metrics that must not significantly worsen. If the primary metric improves but a guard metric degrades, the experiment fails.

Example:

• Primary: 7-day retention rate
• Guards: session length (must not drop > 5%), support ticket volume (must not rise > 10%)

Step 3: Calculate required sample size before running

Use a power calculation to determine how many observations you need. Running underpowered experiments wastes time and produces false negatives.

Inputs needed:

• Baseline rate: current value of the primary metric (e.g., 22% 7-day retention)
• Minimum detectable effect (MDE): smallest change that would be meaningful (e.g., 2 percentage points)
• Statistical significance level: alpha = 0.05 (5% false positive rate)
• Statistical power: 1 - beta = 0.80 (80% probability of detecting a true effect)

from statsmodels.stats.power import NormalIndPower

analysis = NormalIndPower()
n = analysis.solve_power(
    effect_size=0.02 / (0.22 * (1 - 0.22)) ** 0.5,
    alpha=0.05,
    power=0.80,
    alternative="two-sided"
)
print(f"Required n per group: {n:.0f}")

If the required sample size exceeds your available traffic within a reasonable window, increase the MDE or extend the timeline. Do not run an underpowered test.

Step 4: Design the randomization and control

Randomization unit must match the analysis unit.

• If analyzing per-user metrics: randomize by user ID
• If analyzing per-session metrics: randomize by session ID
• If analyzing geographic effects: randomize by region (cluster randomization)

Document:

• How users are assigned (hash of user ID mod 2, for example)
• Whether assignment is sticky (user always sees the same variant)
• Exclusion criteria (new users only? logged-in only? specific device types?)

Run a sanity check (A/A test) before the real experiment if possible - confirm the two groups are balanced on baseline characteristics.

Step 5: Set the stopping rule before launch

Never stop an experiment early because the results look good. Peeking inflates false positive rates.

Define in advance:

• Minimum runtime: run for at least 1 full business cycle (usually 1-2 weeks) to account for day-of-week effects
• Maximum runtime: set a hard end date
• Sample size target: stop when you hit the calculated N, not when the p-value crosses 0.05

If early stopping is operationally required, use a sequential testing method (e.g., always-valid p-values, group sequential design) instead of standard frequentist stopping rules.

Step 6: Analyze and report results with effect size, not just p-values

Spiegelhalter's emphasis: p-values tell you whether an effect exists; effect sizes tell you whether it matters.

Report:

• Observed difference (absolute and relative)
• 95% confidence interval for the difference
• p-value (as supporting evidence, not the conclusion)
• Practical significance: is the effect size large enough to act on?

from scipy import stats
import numpy as np

control = df[df["group"] == "control"]["converted"]
treatment = df[df["group"] == "treatment"]["converted"]

t_stat, p_value = stats.ttest_ind(control, treatment)
effect_size = treatment.mean() - control.mean()
ci = stats.t.interval(0.95, len(df)-2, loc=effect_size, scale=stats.sem(np.concatenate([control, treatment])))

print(f"Effect: {effect_size:.4f} ({effect_size/control.mean()*100:.1f}% relative lift)")
print(f"95% CI: ({ci[0]:.4f}, {ci[1]:.4f})")
print(f"p-value: {p_value:.4f}")

State the conclusion as: "The data [supports / does not support] rejecting the null hypothesis. The observed effect was [X], with a 95% CI of [Y to Z]. This [is / is not] practically significant because [reason]."

Anti-Patterns

1. Peeking and stopping early Bad: Checking results daily and stopping as soon as p < 0.05. Good: Pre-commit to a sample size and stop date. Do not look at significance until the stopping criterion is met.

2. Multiple primary metrics Bad: "We'll call it a win if conversion OR retention OR revenue improves." Good: One primary metric. Guard metrics are binary pass/fail, not additional wins.

3. Reporting only p-values Bad: "The result was statistically significant (p = 0.03), so we should ship." Good: "The result was significant (p = 0.03). The absolute lift was 0.4pp (95% CI: 0.1 to 0.7). This is below our 1pp MDE threshold, so we will not ship."

4. Ignoring novelty effects Bad: Running an experiment for 3 days and seeing a spike from users trying something new. Good: Running for at least 1-2 weeks and examining whether the effect size stabilizes over time.

Quality Checklist

• [ ] Hypothesis stated in falsifiable form with intervention, population, metric, and direction
• [ ] Single primary metric defined, plus guard metrics with acceptable thresholds
• [ ] Sample size calculated using power analysis before the experiment launches
• [ ] Randomization unit matches analysis unit
• [ ] Stopping rule (runtime and sample size target) defined before launch
• [ ] Results reported with absolute effect, relative effect, and 95% confidence interval
• [ ] Practical significance assessed separately from statistical significance