qa-aman/experiment-design

Overview

Based on The Lean Startup by Eric Ries. The core insight: startups (and product teams) don't fail from lack of execution - they fail from executing on the wrong plan. The build-measure-learn loop is how you find the right plan faster. Every feature is a hypothesis. Every release is an experiment. Validated learning - not shipped features - is the unit of progress.

Workflow

Step 1: State the hypothesis

Format: "We believe [action/feature] will cause [user behavior change], resulting in [metric change], because [assumption]."

Example: "We believe adding a progress bar to onboarding will cause more users to complete setup, resulting in a 15% increase in activation rate, because users abandon when they can't see how much is left."

The "because" is critical - it forces you to name the assumption you're testing.

Step 2: Identify the riskiest assumption

Most hypotheses have multiple assumptions embedded. Find the one that, if wrong, makes the whole hypothesis collapse.

List assumptions - rank by risk - test the riskiest one first.

Step 3: Design the minimum viable test

What's the cheapest, fastest way to test the riskiest assumption?

Ries's ladder of experiment types (least to most expensive):

1. Customer interview - ask 5 users if the problem exists
2. Fake door / 404 test - show the feature, measure click rate before building
3. Concierge MVP - do the job manually for a few users
4. Wizard of Oz - appear automated, run manually behind the scenes
5. A/B test - build both versions, split traffic
6. Full release - ship to everyone

Start as low on the ladder as possible. Only move up when the cheaper test is insufficient.

Step 4: Define success criteria before running

Set the threshold before you see results. This prevents moving the goalposts.

"We will consider this experiment successful if [metric] changes by [amount] within [time period] with [sample size]."

Step 5: Calculate required sample size

For A/B tests: use a significance calculator. Undersized tests produce false results.

• Minimum detectable effect: what's the smallest change that matters?
• Baseline conversion rate: what's the current rate?
• Statistical significance: 95% standard (p < 0.05)

Step 6: Run the experiment and measure

• Don't peek at results early (peeking inflates false positives)
• Measure the primary metric and guard metrics
• Document what you observe, not what you expected

Step 7: Extract validated learning

Whatever the result:

• Confirmed - assumption was right, proceed with confidence
• Refuted - assumption was wrong, pivot or kill
• Inconclusive - sample too small or test too short, iterate

Ries: a failed experiment that teaches you something is more valuable than a successful one that teaches you nothing.

Anti-Patterns

1. HiPPO-driven decisions instead of experiments Bad: "The CEO thinks we should add X, so we're adding X." Good: "The CEO thinks X will increase retention. Let's run a 2-week experiment with 10% of users."

2. Moving the goalposts Bad: Setting a 20% lift as success, seeing 8%, calling it a win anyway. Good: Success criteria set before the experiment runs. Results are binary: met or not met.

3. Testing too many variables at once Bad: Redesigning the entire onboarding and A/B testing it. Good: Change one variable per experiment. Otherwise you can't attribute the result.

4. Skipping the concierge step Bad: Building a full automated feature to test an unvalidated assumption. Good: Do it manually for 10 users first. If the manual version doesn't work, the automated one won't either.

Quality Checklist

• [ ] Hypothesis follows "We believe X will cause Y because Z" format
• [ ] Riskiest assumption identified explicitly
• [ ] Minimum viable test chosen (as low on the ladder as possible)
• [ ] Success criteria defined before running
• [ ] Sample size calculated for quantitative tests
• [ ] Guard metrics defined (what must not get worse)
• [ ] Validated learning documented regardless of outcome