Memo 004: Backtest sanity checks
2025-02-12
Simple sanity checks we run on every strategy backtest before adding to the database.
Before a strategy gets a page in the database, we run a short checklist. Nothing fancy, just backtest sanity.
- Out-of-sample or walk-forward: At least one period (e.g. last 2–5 years) was never used for parameter choice. See walk-forward.
- Transaction costs: We assume at least 5–10 bps per side for liquid instruments; more for less liquid. If the strategy doesn’t survive 2x that cost, we note it. See transaction-costs.
- Overfitting check: How many parameters did we tune? If we optimized 10 things on the same data, we treat the result as illustrative, not robust. See overfitting.
- Failure modes: We write down when and why the strategy can fail. If we can’t articulate that, we don’t add it yet.
Strategies like Dual Momentum Equity, Trend Following (Futures), and Cross-Asset Momentum have been through this. We’ll keep updating as we add more checks.
What would change our mind? If we adopt a formal Bayesian or ML-based backtest framework that’s still interpretable, we’d add it to this checklist and document it in the Concepts section.