backtest-expert

---
name: backtest-expert
description: Use when developing, stress-testing, or validating a systematic trading strategy's backtest for robustness, parameter sensitivity, or overfitting.
category: Strategy R&D
version: 1.0.0
tools: []
---

# Backtest Expert

A systematic methodology for evaluating trading strategies that prioritizes robustness over optimistic paper results.

## Core Philosophy
**Goal**: find strategies that "break the least," not strategies that "profit the most" on paper. **Principle**: add friction, stress-test assumptions, and see what survives — a strategy that holds up under pessimistic conditions is more likely to work in live trading.

## When to Use
Developing or validating systematic trading strategies; assessing whether an idea is robust enough for live implementation; troubleshooting a possibly-misleading backtest; learning proper backtesting methodology; avoiding curve-fitting, look-ahead bias, and survivorship bias; assessing parameter sensitivity and regime dependence; setting realistic slippage/execution-cost expectations.

## Workflow

**1. State the hypothesis.** Define the edge in one sentence (e.g. "Stocks that gap up >3% on earnings and pull back to the previous day's close within the first hour offer a mean-reversion opportunity"). If the edge can't be stated clearly, don't proceed to testing.

**2. Codify rules with zero discretion.** Specify entry (exact conditions, timing, price type), exit (stop loss, profit target, time-based), position sizing (fixed $, % of portfolio, volatility-adjusted), filters (market cap, volume, sector, volatility), and eligible universe. No subjective judgment allowed — every decision must be rule-based and unambiguous.

**3. Run an initial backtest.** Minimum 5 years, preferably 10+, across multiple market regimes (bull, bear, high/low volatility), with realistic commissions plus conservative slippage. Check basic viability before investing further time; iterate the hypothesis if fundamentally broken.

**4. Stress-test the strategy — spend 80% of total testing time here.**
- *Parameter sensitivity*: vary the stop loss across 50%/75%/100%/125%/150% of baseline and the profit target across 80%/90%/100%/110%/120%; shift entry/exit timing by ±15-30 minutes. Look for plateaus of stable performance, not narrow spikes — profitability across a stop-loss range of 1.5%-3.0% signals genuine edge, while profitability only at exactly 2.13% signals curve-fitting.
- *Execution friction*: raise slippage to 1.5-2x typical estimates, model worst-case fills (buy at ask+1 tick, sell at bid-1 tick), add order-rejection scenarios, and use pessimistic commission structures.
- *Time robustness*: analyze year-by-year performance, require positive expectancy in the majority of years, and confirm the strategy doesn't rely on 1-2 exceptional periods or a single regime.
- *Sample size*: 30 trades is the absolute floor, 100+ is preferred, 200+ gives high confidence. A 5% per-trade edge needs 100+ trades to distinguish from luck.

**5. Out-of-sample validation via walk-forward analysis.** Optimize on a training period (e.g. years 1-3), test on a validation period (year 4), then roll forward and repeat, comparing in-sample to out-of-sample performance. Warning signs: out-of-sample performance under 50% of in-sample, frequent need to re-optimize parameters, or parameters shifting dramatically between periods.

**6. Evaluate and decide.** Ask whether the edge survives pessimistic assumptions, whether performance is stable across parameter variations, whether it works across multiple regimes, whether the sample size is sufficient, and whether results are realistic rather than "too good to be true." Score across five dimensions — Sample Size, Expectancy, Risk Management, Robustness, and Execution Realism — flag red flags, and reach a verdict: **Deploy** (survives all stress tests with acceptable performance), **Refine** (sound core logic, needs parameter adjustment), or **Abandon** (fails stress tests or rests on fragile assumptions).

## Key Testing Principles
**Punish the strategy** — layer in higher-than-real commissions, 1.5-2x slippage, worst-case fills, order rejections, and partial fills; strategies that survive pessimistic assumptions tend to outperform live. **Seek plateaus, not peaks** — a stable range of parameter values beats one narrow optimum. **Test all cases, not cherry-picked ones** — evaluate every instrument meeting the criteria, including failures, not just hand-picked winners (selective examples create survivorship bias). **Separate idea generation from validation** — intuition is fine for generating hypotheses, but validation must be purely data-driven; never let attachment to an idea color interpretation of results.

## Common Failure Patterns
Parameter sensitivity (only works at exact values); regime-specificity (great in some years, terrible in others); slippage sensitivity (unprofitable once realistic costs are added); too-small sample size; look-ahead bias (results that are "too good to be true"); over-optimization (many parameters, weak out-of-sample results). Recognizing these early saves time before deeper analysis.

## Critical Reminders
Allocate roughly 20% of effort to generating ideas and 80% to trying to break them. A strategy that needs "perfect context" to work isn't robust enough for systematic trading. Treat >90% win rates, minimal drawdowns, or suspiciously perfect timing as a red flag demanding a look-ahead-bias/data-integrity audit. Understand the backtesting platform's own quirks (interpolation methods, low-liquidity handling, data-alignment issues), since these can silently distort results.

## Discretionary vs. Systematic
This methodology is for **systematic/quantitative** backtesting — rules codified in advance, no discretion or "feel" in execution, testing against every historical example rather than cherry-picked cases, with news/macro context deliberately stripped out. Discretionary traders, who rely on subjective judgment, study differently and this methodology may not transfer directly to their process.