jmsktm/A/B Test Designer

A/B Test Designer

Design rigorous A/B tests that produce actionable, statistically significant results. This skill combines experimentation methodology with marketing intuition to help you test the right things, measure correctly, and make confident decisions based on data.

Most A/B tests fail before they start due to poor design: wrong sample sizes, multiple variable contamination, or testing low-impact elements. This skill provides the scientific framework for hypothesis formation, test design, sample size calculation, and result interpretation that separates real insights from statistical noise.

Essential for growth marketers, product managers, CRO specialists, and data-driven teams optimizing conversion funnels.

Core Workflows

Workflow 1: Test Hypothesis Development

1. Data Analysis - Review existing performance metrics
2. Opportunity Identification - Find high-impact test areas
3. Hypothesis Formation - "If we [change], then [outcome] because [rationale]"
4. Success Metric Definition - Primary and secondary KPIs
5. Hypothesis Prioritization - Rank by potential impact
6. ICE Scoring - Impact, Confidence, Ease framework
7. Test Roadmap - Sequence tests strategically

Workflow 2: Test Design & Setup

1. Variable Isolation - Test one thing at a time
2. Control Definition - Current version as baseline
3. Variant Creation - Design the challenger
4. Sample Size Calculation - Required visitors for significance
5. Test Duration Planning - Account for traffic and cycles
6. Segmentation Strategy - Define audience splits
7. Technical Implementation - Testing tool configuration

Workflow 3: Statistical Analysis

1. Significance Threshold - Set confidence level (95% typical)
2. Minimum Detectable Effect - What lift would matter?
3. Power Analysis - Reduce false negative risk
4. P-Value Interpretation - Understand what it means
5. Confidence Intervals - Range of likely outcomes
6. Segment Analysis - Performance by audience
7. Result Documentation - Clear winner/loser/inconclusive

Workflow 4: Multivariate Testing

1. Element Selection - Choose factors to test
2. Combination Matrix - Map all variations
3. Traffic Requirements - Calculate needed sample size
4. Interaction Effects - Look for element synergies
5. Fractional Factorial - Reduce combinations if needed
6. Winner Identification - Best performing combination
7. Learning Extraction - Insights beyond the winner

Quick Reference

| Action | Command/Trigger | |--------|-----------------| | Design test | "Create A/B test for [element/page]" | | Write hypothesis | "Form hypothesis for testing [change]" | | Calculate sample size | "How much traffic do I need to test [change]?" | | Analyze results | "Interpret these A/B test results" | | Prioritize tests | "Prioritize these test ideas using ICE" | | MVT design | "Design multivariate test for [elements]" | | Segment analysis | "Break down results by [segment]" | | Test roadmap | "Create 90-day testing roadmap" |

Best Practices

• Hypothesis first - No hypothesis, no learning
• One variable - Test single changes for clean insights
• Primary metric - Choose one main success measure
• Statistical significance - 95% confidence minimum
• Don't peek - Wait for full sample size
• Consider business cycles - Week over week variation matters
• Document everything - Build testing knowledge base
• Size for significance - Under-powered tests waste time
• High-impact areas first - Test headlines before button colors
• Segment thoughtfully - Look for hidden patterns in data
• Accept null results - "No difference" is valid learning
• Iterate on winners - Compound gains through sequential tests
• Avoid HiPPO - Highest Paid Person's Opinion isn't data
• Share learnings - Tests benefit the whole organization
• Consider externalities - Seasonality, competition, campaigns