Ab Testing

---
name: Ab Testing
description: Use when planning or designing an A/B test/experiment, or building a systematic growth-experimentation program with hypotheses, sample sizing, and ICE prioritization.
category: Paid & Measurement
version: 2.0.0
tools: []
---

# A/B Test Setup

Design experiments that produce statistically valid, actionable results — and build a systematic experimentation program around them.

## Initial Assessment

Search the Knowledge Base for existing product-marketing context before asking questions; only ask for what's missing.

Before designing a test, understand: **test context** — what's being improved and what change is being considered; **current state** — baseline conversion rate, current traffic volume; **constraints** — technical complexity, timeline, tools available.

## Core Principles

Start with a hypothesis, not "let's see what happens" — a specific, reasoned prediction of outcome. Test one variable per test, or the result can't be attributed. Apply statistical rigor: pre-determine sample size, don't peek and stop early, commit to the methodology. Measure what matters: a primary metric tied to business value, secondary metrics for context, guardrail metrics to prevent harm.

## Hypothesis Framework

Structure: "Because [observation/data], we believe [change] will cause [expected outcome] for [audience]. We'll know this is true when [metrics]."

Weak: "Changing the button color might increase clicks."
Strong: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using a contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."

## Test Types

| Type | Description | Traffic needed |
|---|---|---|
| A/B | Two versions, single change | Moderate |
| A/B/n | Multiple variants | Higher |
| MVT (multivariate) | Multiple changes in combination | Very high |
| Split URL | Different URLs per variant | Moderate |

## Sample Size

| Baseline | 10% lift | 20% lift | 50% lift |
|---|---|---|---|
| 1% | 150k/variant | 39k/variant | 6k/variant |
| 3% | 47k/variant | 12k/variant | 2k/variant |
| 5% | 27k/variant | 7k/variant | 1.2k/variant |
| 10% | 12k/variant | 3k/variant | 550/variant |

Use a sample-size calculator (Evan Miller's or Optimizely's) to check the exact figure for a given baseline and minimum detectable effect before launch.

## Metrics Selection

Primary metric: the single metric that matters most, directly tied to the hypothesis, used to call the test. Secondary metrics support interpretation of the primary. Guardrail metrics are things that shouldn't get worse — stop the test if one moves significantly negative. Example, a pricing-page test: primary = plan selection rate; secondary = time on page, plan distribution; guardrail = support tickets, refund rate.

## Designing Variants and Traffic Allocation

Vary headlines/copy (message angle, value prop, specificity, tone), visual design (layout, color, images, hierarchy), CTA (button copy, size, placement, count), or content (information included, order, amount, social proof) — one meaningful, bold, hypothesis-true change per test.

Traffic split: 50/50 standard for A/B; 90/10 or 80/20 conservative to limit risk of a bad variant; ramping (start small, increase) for technical risk mitigation. Keep exposure consistent (same variant on return visits) and balanced across time of day/week.

## Implementation

Client-side (JavaScript modifies the page after load — quick, can cause flicker; PostHog, Optimizely, VWO) vs. server-side (variant determined before render — no flicker, requires dev work; PostHog, LaunchDarkly, Split).

## Running the Test

Pre-launch: hypothesis documented, primary metric defined, sample size calculated, variants implemented correctly, tracking verified, QA'd on all variants. During the test, monitor for technical issues, check segment quality, document external factors — and avoid peeking/stopping early, changing variants mid-test, or adding new traffic sources.

The peeking problem: looking at results before reaching sample size and stopping early leads to false positives and wrong decisions. Pre-commit to sample size and trust the process.

## Analyzing Results

Statistical significance: 95% confidence = p-value < 0.05, meaning less than a 5% chance the result is random — a threshold, not a guarantee.

Analysis checklist: reached sample size (if not, the result is preliminary); statistically significant (check confidence intervals); effect size meaningful (compare to the minimum detectable effect, project impact); secondary metrics consistent with the primary; any guardrail concerns; segment differences (mobile vs. desktop, new vs. returning).

| Result | Conclusion |
|---|---|
| Significant winner | Implement the variant |
| Significant loser | Keep control, learn why |
| No significant difference | Need more traffic or a bolder test |
| Mixed signals | Dig deeper, maybe segment |

Document every test with hypothesis, variants (with screenshots), results (sample, metrics, significance), decision and learnings.

## Growth Experimentation Program

A continuous experimentation program compounds value beyond any single test. The experiment loop: generate hypotheses (from data, research, competitors, customer feedback) → prioritize with ICE scoring → design and run the test → analyze with statistical rigor → promote winners to a playbook → generate new hypotheses from learnings → repeat.

**Hypothesis generation sources:** analytics (drop-off points, low-converting pages, underperforming segments); customer research (pain points, confusion, unmet expectations); competitor analysis (features, messaging, UX patterns used elsewhere); support tickets (recurring complaints about conversion flows); heatmaps/recordings (hesitation, rage-clicks, abandonment); past experiments ("significant loser" tests often reveal new angles).

**ICE prioritization:** score each hypothesis 1–10 on Impact (how much it moves the primary metric if it works), Confidence (how sure, based on data not gut), and Ease (how fast/cheap to ship and measure). ICE Score = (Impact + Confidence + Ease) / 3. Run the highest-scoring experiments first; re-score monthly as context changes.

**Experiment velocity** — track as a leading indicator: 4–8 experiments launched per month for most teams; 20–30% win rate is typical for mature programs (sustained higher rates may signal overly conservative hypotheses); 2–4 week average test duration; 20+ hypotheses queued in the backlog; cumulative lift compounding from all winners.

**The experiment playbook.** When a test wins, document the pattern, not just the implementation: experiment name, date, hypothesis, sample size, result (winner/loser/inconclusive, primary metric change with 95% CI and p-value), guardrail outcomes, segment deltas, why it worked or failed, the reusable pattern (e.g. "social proof near pricing CTAs increases plan selection"), where else it might apply, and status (implemented/parked/needs follow-up). Over time this becomes a library of proven growth patterns specific to the product and audience.

**Cadence:** weekly (30 min) — review running experiments for technical issues and guardrail metrics, stop tests with significantly negative guardrails but don't call winners early. Bi-weekly — conclude completed experiments, update the playbook, launch the next from the backlog. Monthly (1 hour) — review velocity, win rate, cumulative lift; replenish the backlog; re-prioritize with ICE. Quarterly — audit the playbook for patterns applied broadly, winning patterns not yet scaled, and under-tested funnel areas.

## Common Mistakes

Design: too small a change (undetectable), too many things at once (can't isolate the cause), no clear hypothesis. Execution: stopping early, changing variants mid-test, not checking implementation. Analysis: ignoring confidence intervals, cherry-picking segments, over-interpreting inconclusive results.

## Task-Specific Questions

1. Current conversion rate and page traffic volume?
2. What change is being considered, and why?
3. Smallest improvement worth detecting?
4. Testing tools available, and has this area been tested before?