experiment-designer

---
name: experiment-designer
description: Use when planning product experiments, writing testable hypotheses, estimating sample size, prioritizing tests, or interpreting A/B outcomes with practical statistical rigor.
category: Product
version: 1.0.0
tools: []
---

# Experiment Designer

Design, prioritize, and evaluate product experiments with clear hypotheses and defensible decisions.

## When To Use

- A/B and multivariate experiment planning
- Hypothesis writing and success criteria definition
- Sample size and minimum detectable effect planning
- Experiment prioritization with ICE scoring
- Reading statistical output for product decisions

## Core Workflow

1. Hypothesis in If/Then/Because format: If we change `[intervention]`, then `[metric]` will change by `[expected direction/magnitude]`, because `[behavioral mechanism]`.
2. Define metrics before running the test: one primary decision metric, guardrail metrics for quality/risk protection, secondary metrics for diagnostics only.
3. Estimate sample size from the baseline conversion rate or mean, the minimum detectable effect (MDE, absolute or relative), the significance level (alpha), and statistical power. A larger MDE or looser alpha/power needs a smaller sample; a smaller MDE or stricter alpha/power needs a larger sample. Compute the required sample size per variant and in total before launching.
4. Prioritize with ICE — Impact (potential upside), Confidence (evidence quality), Ease (cost/speed/complexity): ICE Score = (Impact * Confidence * Ease) / 10.
5. Launch with stopping rules: decide fixed sample size or fixed duration in advance, avoid repeated peeking without a proper sequential-testing method, monitor guardrails continuously.
6. Interpret results: statistical significance is not business significance; compare the point estimate plus confidence interval to the decision threshold; investigate novelty effects and segment heterogeneity.

## Hypothesis Quality Checklist

- Explicit intervention and audience
- Measurable metric change specified
- Plausible causal reason stated
- Expected minimum effect included
- Failure condition defined

## Common Experiment Pitfalls

- Underpowered tests leading to false negatives
- Too many simultaneous changes without isolation
- Changing targeting or implementation mid-test
- Stopping early on random spikes
- Ignoring sample ratio mismatch and instrumentation drift
- Declaring success from a p-value without effect-size context

## Statistical Interpretation Guardrails

- p-value < alpha is evidence against the null, not guaranteed truth.
- A confidence interval crossing zero/no-effect means the directional claim is uncertain.
- Wide intervals imply low precision even when significant.
- Use practical significance thresholds tied to business impact.