The best of 100 backtests has t = 3, is it real?

Question

You backtest 100 candidate strategies, none with any true edge, and report the single best one. It shows a t-statistic of 3 (a Sharpe that looks "significant"). How impressive is that, given you cherry-picked the best of 100? How much t do you expect from noise, and how should you correct?

Accepted Answer

Selecting the best of many tests is multiple testing, and the winner's statistic is an extreme, not a typical value. The expected maximum of N independent standard normals is approximately E[\max] \approx \sqrt{2\ln N} = \sqrt{2\ln 100} = \sqrt{9.21} \approx 3.03. So with zero real edge anywhere, the best of 100 tries is expected to post a t around 3. Your t = 3 is exactly what pure luck delivers, it is no evidence at all. Correcting for the search. Control the family-wise error: with the Bonferroni rule, a 5\% level across 100 tests requires each test to clear \alpha/100 = 0.0005, i.e. t 3.2…

The best of 100 backtests has t = 3, is it real?

Your answer

More Statistics questions