Why the best backtest is probably a fluke

Question

A researcher tries 200 strategy variants on the same price history and reports the one with the highest backtested Sharpe ratio. Why is that reported Sharpe likely to overstate reality, and how would you correct for it?

Accepted Answer

The problem is multiple testing (also called selection bias or backtest overfitting). Even if none of the 200 strategies has any real edge, their backtested Sharpes will scatter around zero just from noise, and the maximum of 200 noisy numbers is well above zero. By reporting only the winner, you are reporting an extreme of a random sample, so its Sharpe is inflated by the very act of picking the best. The more variants you try, the higher the best one looks by luck alone. This is the finance cousin of p-hacking: search a big enough space of rules, parameters, and universes and something will…

Why the best backtest is probably a fluke

Your answer

More Statistics questions