ab-test-setup

---
name: ab-test-setup
description: Use when the user wants to plan, design, or implement an A/B test or experiment — hypothesis, sample size, variants, statistical significance, or test result interpretation.
category: Marketing
version: 1.0.0
tools: []
---

# A/B Test Setup

Design experiments that produce statistically valid, actionable results — from hypothesis through sample-size calculation, variant design, and result interpretation.

## Initial Assessment
Before designing a test, understand: **Test Context** (what's being improved, what change is being considered), **Current State** (baseline conversion rate, traffic volume), and **Constraints** (technical complexity, timeline, tools available). If prior product/marketing context is available, use it and only ask about gaps.

## Core Principles
1. **Start with a hypothesis** — a specific, reasoned prediction, not "let's see what happens"
2. **Test one thing** — a single variable per test, or you won't know what worked
3. **Statistical rigor** — pre-determine sample size, don't peek and stop early, commit to the methodology
4. **Measure what matters** — a primary metric tied to business value, secondary metrics for context, guardrail metrics to catch harm

## Hypothesis Framework
Structure: "Because [observation/data], we believe [change] will cause [expected outcome] for [audience]. We'll know this is true when [metrics]."

Weak: "Changing the button color might increase clicks."
Strong: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and higher-contrast will increase CTA clicks by 15%+ for new visitors, measured as click-through from page view to signup start."

## Test Types
| Type | Description | Traffic Needed |
|---|---|---|
| A/B | Two versions, single change | Moderate |
| A/B/n | Multiple variants | Higher |
| MVT | Multiple changes in combination | Very high |
| Split URL | Different URLs per variant | Moderate |

## Sample Size
Calculate sample size explicitly — don't eyeball it — using a two-proportion z-test at α=0.05 (two-tailed) and 80% power against a relative minimum detectable effect (MDE). Quick reference (samples needed per variant):

| Baseline | 10% Lift | 20% Lift | 50% Lift |
|---|---|---|---|
| 1% | 163k | 43k | 7.7k |
| 3% | 53k | 14k | 2.5k |
| 5% | 31k | 8.2k | 1.5k |
| 10% | 15k | 3.8k | 683 |

Record the required sample size and estimated test duration (sample size ÷ daily eligible traffic) in the test plan's "sample size + duration" line before any test is approved to run. Cross-check against a public calculator (e.g. Evan Miller's or Optimizely's sample-size calculators) if precision matters.

## Metrics Selection
- **Primary**: the single metric tied directly to the hypothesis — what decides the test
- **Secondary**: metrics that explain why/how the change worked
- **Guardrail**: metrics that must not get worse; stop the test if they move significantly negative

Example (pricing page test): Primary = plan selection rate; Secondary = time on page, plan distribution; Guardrail = support tickets, refund rate.

## Designing Variants
Vary one meaningful thing: headlines/copy (message angle, value prop, tone), visual design (layout, color, hierarchy), CTA (copy, size, placement), or content (information included, order, social proof). The change should be bold enough to matter and true to the hypothesis.

## Traffic Allocation
| Approach | Split | When |
|---|---|---|
| Standard | 50/50 | Default for A/B |
| Conservative | 90/10, 80/20 | Limit risk of a bad variant |
| Ramping | Start small, increase | Mitigate technical risk |

Keep allocation consistent per user across visits, and balance exposure across time of day/week.

## Implementation
**Client-side** (JS modifies page after load): quick, can cause flicker; tools like PostHog, Optimizely, VWO. **Server-side** (variant decided before render): no flicker, requires dev work; tools like PostHog, LaunchDarkly, Split.

## Running the Test
Pre-launch checklist: hypothesis documented, primary metric defined, sample size calculated, variants implemented correctly, tracking verified, QA completed on all variants.

During the test: do monitor for technical issues, check segment quality, document external factors. Don't peek and stop early, don't change variants mid-test, don't add new traffic sources. Stopping early or peeking before reaching the pre-committed sample size produces false positives — commit to the sample size and trust the process.

## Analyzing Results
Statistical significance at 95% confidence means p < 0.05 — under 5% chance the result is random, not a guarantee. Work through: did it reach sample size (otherwise the result is preliminary)? Is it statistically significant (check confidence intervals)? Is the effect size meaningful relative to the MDE? Do secondary metrics support the primary? Did any guardrail move negative? Are there meaningful segment differences (mobile vs. desktop, new vs. returning)?

| Result | Conclusion |
|---|---|
| Significant winner | Implement variant |
| Significant loser | Keep control, learn why |
| No significant difference | Need more traffic or a bolder test |
| Mixed signals | Dig deeper, consider segmenting |

## Documentation
Document every test with: hypothesis, variants (described/screenshotted), results (sample size, metrics, significance), and the decision plus learnings.

## Common Mistakes
**Design**: too small a change to detect, too many things tested at once, no clear hypothesis. **Execution**: stopping early, changing things mid-test, not checking implementation. **Analysis**: ignoring confidence intervals, cherry-picking segments, over-interpreting inconclusive results.

## Proactive Triggers
Offer to design a test when: a conversion rate is shared and improvement is being discussed; two copy/design variants are being debated instead of tested; a campaign or landing page underperforms; pricing-page changes come up; or after a feature/campaign launch, as a follow-up experiment.

## Output Artifacts
Experiment Brief (hypothesis, variants, metrics, sample size, duration, owner), Sample Size Calculator Input table, Pre-Launch QA Checklist, Results Analysis Report, and a prioritized Test Backlog.

Every test should produce a learning even when the variant loses — avoid presenting inconclusive results as wins. If the work overlaps a teammate's specialty (what to test, tracking infrastructure, pricing strategy, broader campaign attribution), @mention them rather than guessing at their part.