Scale Subject Line A/B Testing with Statistical Analysis

Create a scale subject line A/B testing program for [COMPANY/AGENCY NAME], responsible for [NUMBER] sends/month with a database of [NUMBER] contacts.

**Context:**
- Platform: [ACTIVECAMPAIGN/KLAVIYO/MAILCHIMP/HUBSPOT]
- Current average open rate: [PERCENTAGE]%
- Industry benchmark: [PERCENTAGE]%
- Sending frequency: [NUMBER]/month
- Available segments: [LIST]

**1) Statistical foundations (simplified for practical use):**

*Minimum sample size:*
- To detect a 10% improvement in opens with 95% confidence:
  - Required base per variant: approximately [NUMBER] contacts
  - Formula: use the Optimizely or Evan Miller calculator
  - For smaller bases: accept 85% confidence (faster, less precise)

*When to declare a winner:*
- Wait minimum of [HOURS] after sending (most opens occur within [X] hours)
- Minimum confidence: 85% (operational) or 95% (strategic decision)
- If inconclusive: record as inconclusive, do not force a winner

*Variable isolation:*
- NEVER test more than 1 element at a time
- Same send time, same segment, same email content

**2) Hypothesis library to test (prioritized backlog):**

Ordered by potential impact (highest first):

**High priority (highest expected impact):**
1. Personalization with name vs. without: "[NAME], have you seen this?" vs. "Have you seen this?"
2. Question vs. statement: "How do you [X]?" vs. "Learn to [X]"
3. Short (<30 chars) vs. long (50+ chars): for mobile (70% opens on phone)
4. With emoji vs. without: in initial vs. final position
5. Specific number vs. generic: "7 ways" vs. "Several ways"

**Medium priority:**
6. Explicit vs. implicit urgency: "Only until today" vs. "This offer is for a limited time"
7. Benefit vs. curiosity: "Win [X]" vs. "Did you know..."
8. Lowercase vs. standard capitalization: "this changed everything" vs. "This Changed Everything"
9. Sender: company name vs. person's name
10. Preview text: with vs. without optimized preview text

**Low priority:**
11. With vs. without exclamation mark
12. Preview text length
13. Use of brackets [ACTION]
14. Written numbers vs. numerals

**3) Test documentation template:**

```
=== TEST #[NUMBER] ===
Send date: [DATE]
Segment: [SEGMENT] ([NUMBER] total contacts)
Split: 50% A / 50% B (or [X]% A / [X]% B for large base)

Hypothesis: "If [CHANGE], we expect [RESULT] because [REASON]"
Variable tested: [EXACTLY WHAT CHANGES]

Variant A (control):
  Subject line: "[TEXT]"
  Preview text: "[TEXT]"
  Sender: [NAME]

Variant B (challenger):
  Subject line: "[TEXT]"
  Preview text: "[TEXT]"
  Sender: [NAME]

Results (after [X] hours):
  A: Open rate: [X]% | Clicks: [X]%
  B: Open rate: [X]% | Clicks: [X]%
  Difference: [X] p.p. ([X]% improvement)
  Statistical confidence: [X]%

Winner: [A / B / INCONCLUSIVE]
Effect size: [SMALL <5% / MEDIUM 5-15% / LARGE >15%]

Learning: [What this teaches us BEYOND this email]
Action: [What we change in our process going forward]
Next suggested test: [DERIVED HYPOTHESIS]
```

**4) Learnings knowledge base (update after each test):**

Progressive table format:

| Element | What works | What does NOT work | Confidence | Applicable contexts |
|---------|------------|--------------------|-----------|---------------------|
| Personalization | Name in subject +8% opens | Name in subject for promos (looks like spam) | High (12 tests) | Newsletter, content |
| Emojis | Emoji at start +5% mobile | More than 1 emoji (looks like spam) | Medium (4 tests) | E-commerce, special dates |
| ... | | | | |

**5) Testing cadence by sending frequency:**

*For senders with 4+ emails/month:*
- 1 test per week (4 tests/month)
- 12-week cycle: test 1 hypothesis at a time
- Quarterly review: which learnings become policy?

*For senders with 1-3 emails/month:*
- 1 test per email sent (no exceptions)
- Monthly analysis of accumulated results

**6) Cross-context analysis:**

How to learn from one segment and apply to others:
- Successful test in newsletter → test in automation
- Successful test in promotion → test in content
- Test with audience [A] → test with audience [B]

**7) Testing program performance dashboard:**

| Metric | Quarter 1 | Quarter 2 | Growth |
|--------|-----------|-----------|--------|
| Tests conducted | | | |
| % with conclusive result | | | |
| Average open rate improvement | | | |
| Baseline open rate | | | |
| Current open rate | | | |
| Total uplift | | | |

Goal: increase open rate from [X]% to [Y]% in [Z] quarters through systematic testing.