coalesce-labs/experiment-decision

Experiment Decision Framework: When to A/B Test vs Ship

Quick Start

/experiment-decision

Then provide:

1. What you're considering building (feature, change, or experiment)
2. Expected impact (metric + estimated improvement)
3. Your concern (is this risky? reversible? controversial?)

I'll walk you through the decision tree: reversibility, hypothesis strength, detectable impact, and risk level. You'll get a clear recommendation: A/B test, ship + monitor, or just ship.

Output: Decision documented inline or saved to thoughts/shared/product/decisions/ Time: ~5 min for clear-cut cases, ~15 min for nuanced decisions

When to use: Before building any feature, when stakeholders demand "data-driven" decisions, or when unsure if testing is worth the effort

Framework source: Aakash Gupta's "When to A/B Test vs Just Ship"

---

The Decision Framework

Use this decision tree:

Question 1: Is it reversible?

If YES → Ship it

• CSS changes
• Messaging tweaks
• UI polish
• Non-destructive features

Why: Reversible changes have low risk. Ship, monitor, rollback if needed.

If NO → Continue to Question 2

---

Question 2: Do you have a hypothesis with measurable impact?

If NO → Don't test

• Building "nice to haves"
• No clear success metric
• Can't measure the outcome

Why: Testing without a hypothesis is wasteful. Either clarify the hypothesis or don't build it.

If YES → Continue to Question 3

---

Question 3: Is the expected impact large enough to detect?

Run a power calculation:

Minimum Detectable Effect (MDE) = Effect you need to see to justify the work

If your feature is expected to improve conversion by 0.5%, but you need 10M users to detect it → Don't test, just ship and monitor

If impact is too small to detect → Ship without test

If impact is detectable → Continue to Question 4

---

Question 4: Is the risk of being wrong high?

High risk scenarios:

• Affects revenue directly (pricing, checkout)
• Impacts core user experience (onboarding, core flows)
• Controversial decision (stakeholder disagreement)
• Large engineering investment

If HIGH risk → A/B test

If LOW risk → Ship without test

---

Decision Matrix

| Risk Level | Impact Size | Reversible? | Decision | | ---------- | ----------- | ----------- | --------------------------------------- | | High | Large | No | A/B Test | | High | Large | Yes | A/B Test (or ship with kill switch) | | High | Small | No | Don't build | | High | Small | Yes | Ship + Monitor | | Low | Large | No | Ship + Monitor | | Low | Large | Yes | Just Ship | | Low | Small | No | Just Ship | | Low | Small | Yes | Just Ship |

---

When to A/B Test

✅ Test When:

1. High-stakes decisions

• Pricing changes
• Checkout flow modifications
• Core product changes
• Revenue-impacting features

2. Controversial hypotheses

• Team is divided on approach
• Stakeholders disagree
• User research is conflicting

3. Long-term bets

• Features that are expensive to reverse
• Architectural decisions
• Platform changes

4. Optimization work

• Conversion rate improvements
• Engagement optimization
• Retention experiments

---

When to Just Ship

✅ Ship When:

1. Fast iteration needed

• Competitive pressure
• Time-sensitive opportunities
• Market windows closing

2. Low risk, high certainty

• Bug fixes
• Obvious improvements
• User-requested features (with clear demand)

3. Qualitative insights are strong

• Clear user pain validated through research
• Competitive parity features
• Accessibility improvements

4. Testing would take too long

• Small user base (can't reach significance)
• Slow conversion cycles (months to convert)
• Complex setup (weeks to build test infrastructure)

---

The Cost of A/B Testing

Time costs:

• Engineering: 2-4 weeks to build test infrastructure
• Analysis: 1-2 weeks to run experiment + analyze
• Total: 3-6 weeks delay

Engineering costs:

• Feature flagging system
• Analytics instrumentation
• A/A test validation
• Test maintenance

Opportunity costs:

• Could have shipped 3-5 other features
• Delayed value delivery to users
• Competitors may ship first

When testing costs exceed value → Just ship

---

Real-World Examples

Example 1: Amazon's "Add to Cart" Button Color

Decision: A/B Test

• High traffic (millions of users)
• Direct revenue impact
• Easy to detect small improvements
• Result: +2% conversion = $100M+ annually

---

Example 2: Slack's Message Threading

Decision: Just Ship

• Highly requested feature
• Strong qualitative signal from users
• Reversible (users can ignore threads)
• Result: Successful launch, became core feature

---

Example 3: Netflix's "Are you still watching?" prompt

Decision: A/B Test

• Controversial (could annoy users)
• Impact on engagement unclear
• Risk of hurting retention
• Result: Test showed improved engagement (prevented zombie sessions)

---

Common Mistakes

❌ Testing everything "to be data-driven"

• Problem: Slows down velocity
• Fix: Reserve tests for high-stakes decisions

❌ Shipping without monitoring

• Problem: Bad changes go unnoticed
• Fix: Ship with dashboards and alerts

❌ Running underpowered tests

• Problem: Waste time on inconclusive results
• Fix: Calculate sample size before starting

❌ Testing when qualitative data is clear

• Problem: Delays obvious improvements
• Fix: Trust strong user research signals

---

Quick Reference Checklist

Before building any feature, ask:

• [ ] Is this reversible? (If yes → ship)
• [ ] Do I have a clear hypothesis? (If no → don't build)
• [ ] Can I measure the impact? (If no → don't test)
• [ ] Is the expected impact large enough to detect? (Power calculation)
• [ ] What's the risk of being wrong? (High risk → test)
• [ ] What's the cost of testing vs shipping? (ROI check)
• [ ] Do I have strong qualitative data? (If yes → consider shipping)

---

Statistical Power Guidance

Before committing to an A/B test, estimate whether you have enough traffic to detect a meaningful difference.

Power Calculation Essentials

Three inputs you need:

1. Minimum Detectable Effect (MDE) -- what's the smallest improvement worth detecting?

   • For checkout conversion: 1-2% relative change matters (high revenue impact)
   • For feature adoption: 5-10% relative change is typical MDE
   • For engagement metrics: 3-5% relative change is reasonable

2. Baseline conversion rate -- what's the current rate you're trying to improve?

   • Higher baselines need more samples to detect small changes
   • Lower baselines are easier to move (but may need larger sample)

3. Daily traffic to the experiment -- how many users will enter the test per day?

Rule of Thumb

You need approximately 1,000 conversions per variant to detect a 5% relative change at 80% power (95% confidence).

| Baseline Rate | MDE (Relative) | Conversions Needed Per Variant | At 1K daily visitors, days needed | | ------------- | -------------- | ------------------------------ | --------------------------------- | | 50% | 5% | ~3,200 | ~7 days | | 20% | 5% | ~12,500 | ~63 days | | 5% | 10% | ~15,000 | ~300 days | | 2% | 10% | ~40,000 | ~800 days |

When Traffic Is Too Low

If your power calculation shows the test would take longer than 4-6 weeks:

1. Accept a larger MDE -- only test if you expect a big swing (15%+ improvement)
2. Use a composite metric -- combine multiple success signals into one metric for higher sensitivity
3. Run a qualitative test -- 5-10 user tests instead of a statistical A/B test
4. Just ship and monitor -- launch with clear success criteria, compare before/after with caveats
5. Use Bayesian methods -- more forgiving with small samples, give probability ranges instead of p-values

Common Pitfalls

• Peeking at results early -- checking before reaching sample size inflates false positive rate. Commit to a runtime upfront.
• Stopping at first significant result -- random fluctuations can look significant early. Use sequential testing if you must peek.
• Testing too many variants -- each variant divides your traffic. Stick to 2-3 variants max.

---

When to Skip the Framework

Some decisions don't need the full decision tree:

1. Regulatory/Compliance Requirement

Action: Just ship it. You don't have a choice. But: Document the change, set up monitoring, track any user impact.

2. Bug Fix

Action: Just fix it. No one A/B tests bug fixes. But: If the "bug fix" changes user behavior significantly, monitor post-fix metrics.

3. CEO/Board Mandate

Action: Document the decision and ship. Set up measurement so you can report on impact. But: Frame your measurement as "proving the impact" rather than "testing whether to do it." This builds credibility for future data-driven decisions.

4. Competitive Response

Action: If a competitor just shipped a similar feature and your users are asking for it, speed matters more than experimentation. Ship fast, measure after. But: Don't use "competitive pressure" as an excuse for every feature. Reserve this for genuine market urgency.

5. Sunset/Deprecation

Action: If you're removing a feature that <1% of users touch, just remove it with advance notice. But: If the feature has any paying customers relying on it, communicate early and provide alternatives.

---

Output Quality Self-Check

Before delivering the experiment decision, verify:

• [ ] Decision is clear -- the recommendation is explicitly "A/B test," "Ship + Monitor," or "Just Ship"
• [ ] Reversibility is assessed with specific reasoning (not just "yes/no")
• [ ] Hypothesis is stated in If/Then/Because format
• [ ] Power calculation is included if recommending a test (MDE, baseline, sample size, duration)
• [ ] Risk level is justified with specific stakes (revenue impact, user count affected)
• [ ] Cost of testing is weighed against cost of being wrong
• [ ] Edge cases are checked (compliance, bug fix, mandate, competitive response)
• [ ] Stakeholder consensus is noted -- does the team agree on the approach?
• [ ] Monitoring plan exists regardless of decision (even "just ship" needs dashboards)
• [ ] Next step is clear -- if testing, what metrics? If shipping, what success criteria?
• [ ] Connected to past decisions -- have we made similar decisions before? What happened?

---

Related Skills

• /experiment-metrics - Choose the right metrics to measure
• /activation-analysis - Test activation improvements
• /metrics-framework - Understand leading vs lagging metrics
• /define-north-star - Align tests to North Star

---

Framework credit: Adapted from Aakash Gupta's experiment decision frameworks. Read: https://www.news.aakashg.com/p/when-to-ab-test

---

Context Routing Strategy

When the PM uses /experiment-decision, I automatically:

1. Check Historical Reversibility Precedent

Source: thoughts/shared/product/decisions/, past decisions

• What I look for: Similar decisions, how reversibility was judged
• How I use it: Ensure consistent reversibility assessment
• Example: "Last time we shipped CSS changes without testing; this is similar"

2. Extract Success Metrics Framework

Source: thoughts/shared/pm/metrics/, active PRDs

• What I look for: What metrics you typically measure, variance patterns
• How I use it: Calculate minimum detectable effect (MDE) more accurately
• Example: "Based on your metrics history, conversion rate variance is 3%, so MDE = 2%"

3. Route to Experiment Metrics if Testing

Source: Connection to /experiment-metrics skill

• What I look for: Whether decision routes to testing
• How I use it: If decision is "test", auto-suggest next step with /experiment-metrics
• Example: "Now that you've decided to test, let's pick the right metrics using STEDII"

4. Check Stakeholder Consensus on Risk

Source: thoughts/shared/pm/context/stakeholder-template.md, recent discussions

• What I look for: Stakeholder risk tolerance, veto power
• How I use it: Surface if high-risk decision needs executive approval
• Example: "CEO is risk-averse, so even medium-risk decisions should be tested"

5. Calculate Cost of Testing vs Shipping

Source: Team capacity, past experiment timelines

• What I look for: How long experiments take, engineering cost
• How I use it: ROI calculation in the framework
• Example: "Last experiment took 3 weeks; if we ship in 1 week and monitor, ROI favors shipping"