athina-ai/messaging-ab-tester

Messaging A/B Tester

Stop debating which message is better — test it. Generate messaging variants, deploy them through real channels, and measure which framing actually resonates with your ICP.

Core principle: At seed/Series A, you don't have enough traffic for website A/B tests. But you do have enough LinkedIn impressions and cold email sends to test messaging angles fast.

When to Use

• "Which of these value props should we lead with?"
• "Test our messaging angles and tell me which works"
• "I can't decide between [message A] and [message B]"
• "What messaging resonates most with [ICP]?"
• "Run a messaging test for [product/feature]"

Phase 0: Intake

What to Test

1. Core value prop — The claim or positioning you want to test (e.g., "We help growth teams run outbound 10x faster")
2. Test goal — What are you deciding? (Headline for website, cold email angle, LinkedIn content strategy, ad copy direction)
3. ICP — Who should this resonate with? (Title, company type, stage)
4. Current messaging — What are you using today? (Baseline to beat)

Test Channel

5. Where to test:
   • LinkedIn organic — Post variants across consecutive days, compare engagement
   • Cold email — A/B test subject lines or opening hooks via Smartlead
   • Both — Run in parallel for fastest signal
6. Sample size available:
   • LinkedIn: followers/typical impressions per post
   • Email: list size available for testing

Constraints

7. Number of variants — 3-5 recommended (more = slower signal)
8. Test duration — How long to run? (Default: 1 week for LinkedIn, 3-5 days for email)

Phase 1: Generate Messaging Variants

Create 3-5 variants that test different angles, not just different words. Each variant should represent a distinct strategic bet:

Variant Types

| Type | What It Tests | Example | |------|--------------|---------| | Outcome-driven | Leading with the result | "3x your pipeline in 30 days" | | Pain-driven | Leading with the problem | "Tired of spending 4 hours a day on manual prospecting?" | | Identity-driven | Leading with who they are | "Built for growth teams who move fast" | | Proof-driven | Leading with evidence | "How [Customer] went from 10 to 50 demos/month" | | Contrast-driven | Leading with what you're not | "Not another CRM. An outbound engine." |

Variant Template

For each variant:

VARIANT [N]: [Type — e.g., "Outcome-driven"]

Hypothesis: This framing will resonate because [reasoning tied to ICP psychology]

LinkedIn post version:
---
[Full post copy — 100-200 words, native LinkedIn format]
---

Email subject line version:
[Subject line — max 50 chars]

Email opening hook version:
[First 2 sentences of an email]

Headline version:
[Website headline — max 10 words]

Phase 2: Deploy Tests

Option A: LinkedIn Organic Test

Setup:

1. Schedule variants as consecutive posts (1 per day, same time of day)
2. Each post should be similar length and format (control for post structure)
3. Don't boost any posts — organic only for clean comparison

Measurement (after 48 hours per post):

• Impressions
• Reactions (likes, celebrates, etc.)
• Comments
• Comment sentiment (positive/negative/neutral)
• Profile visits (if trackable)
• DMs received mentioning the post

Option B: Cold Email A/B Test

Setup via your outreach tool (Smartlead, Instantly, Lemlist, or any tool with A/B testing):

1. Create campaign with all variants as A/B test sequences
2. Split list evenly across variants (minimum 50 per variant for signal)
3. Same send time, same sender, same CTA — only the messaging changes

Measurement (after 5 days):

• Open rate (tests subject line)
• Reply rate (tests full message resonance)
• Positive reply rate (tests conversion quality)
• Click rate (if link included)

Option C: Both (Recommended)

Run LinkedIn and email in parallel. Different channels may show different winners — that's valuable signal about where each message works best.

Phase 2B: Collect Results

After the test has run for the planned duration, gather your results:

How to provide data:

• Paste metrics — Copy open rates, reply rates, engagement numbers directly into the chat
• CSV export — Export campaign analytics from your outreach tool and share the file
• Screenshot — Take a screenshot of your dashboard/analytics and share it
• Manual input — Just tell the agent the numbers: "Variant A got 45% open rate and 3% reply rate, Variant B got 52% open rate and 5% reply rate"

For LinkedIn tests: Go to your post analytics (click "View analytics" on each post) and share impressions, reactions, comments, and profile visits per post.

For email tests: Export or screenshot your campaign's variant/A-B test results showing sends, opens, and replies per variant.

The agent will normalize whatever format you provide into the scoring framework below.

Phase 3: Analyze Results

Scoring Framework

| Metric | Weight (LinkedIn) | Weight (Email) | |--------|-------------------|----------------| | Engagement rate | 30% | — | | Comment quality | 30% | — | | Open rate | — | 30% | | Reply rate | — | 40% | | Positive reply rate | — | 30% | | Impressions | 20% | — | | Profile visits / clicks | 20% | — |

Statistical Significance Check

For email tests:

• Minimum sends per variant: 50 (for directional signal), 200+ (for confident decisions)
• Minimum difference to call a winner: >20% relative difference in primary metric

For LinkedIn tests:

• Minimum posts per variant: 1 (you're testing with limited data — treat as directional)
• Minimum impressions: 500 per post to be comparable

Winner Selection

WINNER: Variant [N] — [Type]

Primary metric: [X] (vs average of [Y] across other variants)
Relative improvement: [Z%] over baseline

Why it won:
[1-2 sentences on what this tells us about ICP messaging preferences]

Runner-up: Variant [N]
[1 sentence on when this might work better — different channel, different segment]

Phase 4: Output Format

# Messaging A/B Test Results — [DATE]
Value prop tested: [description]
ICP: [target audience]
Test duration: [dates]

---

## Test Design

| Variant | Type | Hypothesis |
|---------|------|-----------|
| A | [Type] | [Hypothesis] |
| B | [Type] | [Hypothesis] |
| C | [Type] | [Hypothesis] |

---

## Results

### LinkedIn Test

| Variant | Impressions | Reactions | Comments | Engagement Rate | Score |
|---------|------------|-----------|----------|----------------|-------|
| A | [N] | [N] | [N] | [X%] | [weighted] |
| B | [N] | [N] | [N] | [X%] | [weighted] |
| C | [N] | [N] | [N] | [X%] | [weighted] |

### Email Test

| Variant | Sends | Opens | Open Rate | Replies | Reply Rate | Positive | Score |
|---------|-------|-------|-----------|---------|------------|----------|-------|
| A | [N] | [N] | [X%] | [N] | [X%] | [N] | [weighted] |
| B | [N] | [N] | [X%] | [N] | [X%] | [N] | [weighted] |
| C | [N] | [N] | [X%] | [N] | [X%] | [N] | [weighted] |

---

## Winner: Variant [N] — "[Headline]"

**Why it won:** [Analysis — what does this tell us about how our ICP thinks?]

**Recommended deployment:**
- Website headline: "[adapted version]"
- Sales deck opening: "[adapted version]"
- LinkedIn bio: "[adapted version]"
- Cold email default: "[adapted version]"

---

## Variant Details & Copy

### Variant A: [Full copy used in test]
### Variant B: [Full copy used in test]
### Variant C: [Full copy used in test]

---

## What to Test Next

Based on these results, the next messaging test should explore:
1. [Angle suggested by results — e.g., "test more specific proof points since proof-driven won"]
2. [Segment test — e.g., "test winning message against different ICP segment"]

Save to the current working directory or wherever the user prefers.

Cost

| Component | Cost | |-----------|------| | Variant generation | Free (LLM reasoning) | | LinkedIn posting | Free (organic) | | Email testing | Included with your outreach tool's plan | | Results analysis | Free (LLM reasoning) | | Total | Free |

Tools Required

None. Pure reasoning for variant generation, test design, and result analysis. The user deploys tests through their own tools:

• LinkedIn organic — post variants manually or via scheduling tool
• Cold email — set up A/B tests in whatever outreach tool they use (Smartlead, Instantly, Lemlist, etc.)
• Results — user provides metrics (screenshots, CSV exports, or manual input) for analysis

Trigger Phrases

• "Test which messaging angle works best for [ICP]"
• "Run a messaging A/B test for [value prop]"
• "Which of these messages should we lead with?"
• "Help me decide between these positioning options"