casemark/conducting-backtest-validation

Conducting Backtest Validation

Structures backtesting methodology with out-of-sample testing, cross-validation, and overfitting detection techniques for systematic and factor-based investment strategies.

When To Use

• Validating a new trading strategy or alpha signal before live deployment
• Auditing an existing backtest for overfitting, look-ahead bias, or survivorship bias
• Comparing multiple strategy variants to select a robust candidate
• Reviewing third-party backtest results (fund managers, vendors, research papers)
• Stress-testing a strategy across regime changes, drawdown periods, or tail events

Inputs To Gather

• Strategy specification: signal logic, universe definition, rebalance frequency, position sizing rules, and transaction cost assumptions
• Data sources: price/return series, factor exposures, benchmark indices — confirm whether adjusted for survivorship bias and corporate actions
• Backtest parameters: start/end dates, in-sample vs. out-of-sample split dates, walk-forward window lengths
• Cost model: commissions, slippage estimates, borrowing costs, market impact assumptions [VERIFY against actual execution data if available]
• Benchmark(s): relevant index or factor portfolio for performance attribution
• Prior test count: number of strategy variations already tested on the same dataset (needed for multiple-testing adjustment)

Workflow

1. Data Integrity Audit

• Confirm no look-ahead bias: signals use only information available at decision time
• Check for survivorship bias in the universe (delisted securities, index reconstitution)
• Verify point-in-time correctness for fundamental data (restatements, reporting lags)
• Validate price data for stock splits, dividends, and corporate actions
• Flag any gaps, stale prices, or anomalous returns exceeding ±50% in a single day

2. In-Sample / Out-of-Sample Design

• Split the dataset with a minimum 30% out-of-sample holdout; prefer chronological split over random
• Define walk-forward windows: typical choices are 3–5 year in-sample with 1-year forward test steps
• For shorter histories, use k-fold combinatorial purged cross-validation (CPCV) with an embargo period equal to the strategy's maximum holding period to prevent leakage

3. Overfitting Detection

• Multiple-testing adjustment: apply the Deflated Sharpe Ratio (DSR) or Bonferroni/BHY correction based on the total number of strategy trials
• Parameter sensitivity: vary key parameters ±20% and check whether Sharpe ratio degrades more than 30% — flag fragile strategies
• Minimum Backtest Length (MinBTL): estimate required sample size for statistical significance given observed Sharpe; reject if actual history is shorter [VERIFY formula assumptions against strategy frequency]
• Probability of Backtest Overfitting (PBO): run CPCV and compute the share of OOS combinations that underperform the benchmark — PBO > 0.40 is a red flag

4. Performance & Risk Decomposition

• Report annualized return, volatility, Sharpe ratio, Sortino ratio, max drawdown, and Calmar ratio for both IS and OOS periods
• Decompose returns via factor attribution (market, size, value, momentum, quality at minimum) to isolate residual alpha
• Examine hit rate, profit factor, and average win/loss ratio at the trade level
• Compute turnover and net-of-cost performance; reject strategies where costs consume >50% of gross alpha

5. Regime & Stress Analysis

• Segment performance by market regime: rising rates, falling rates, high vol (VIX > 25), low vol, recession (NBER-dated), expansion
• Identify maximum drawdown duration and recovery period
• Run Monte Carlo reshuffling of trade returns to build confidence intervals around key metrics
• Test sensitivity to execution delay (T+0 vs. T+1 vs. T+2 entry)

6. Replication & Documentation

• Record exact signal definitions, universe filters, and rebalance rules so the backtest is fully reproducible
• Log software version, random seeds, and data vendor/snapshot date
• Archive parameter search space and total trial count for future multiple-testing reference

Output

Produce a Backtest Validation Report containing:

1. Executive summary: strategy description, headline OOS metrics, and pass/fail recommendation
2. Data quality findings: any biases detected, data gaps, or corrections applied
3. IS vs. OOS comparison table: side-by-side metrics with statistical significance notes
4. Overfitting diagnostics: DSR, PBO score, parameter sensitivity heatmap
5. Factor attribution: gross vs. residual alpha, factor loading stability over time
6. Regime analysis: performance table segmented by macro regime
7. Cost impact: gross vs. net Sharpe, breakeven cost threshold
8. Recommendation: deploy, refine, or reject — with specific conditions or thresholds for promotion to paper trading

Quality Checks

• OOS Sharpe ratio is statistically distinguishable from zero at the 95% level (t-stat > 1.96 after multiple-testing adjustment)
• PBO < 0.40 and DSR remains positive after accounting for all trials
• No single regime drives more than 60% of cumulative OOS profit
• Parameter sensitivity analysis shows smooth, not cliff-edge, degradation
• Transaction cost assumptions are realistic — cross-check slippage with actual fill data or TCA reports [VERIFY against broker/execution platform data]
• Factor exposures are stable and intentional; unintended loadings are flagged
• All data sources and methodology steps are documented sufficiently for independent replication