Hansen's Fix: Why Padding Your Search With Bad Strategies Hides the Good One

Halbert White's Reality Check was a genuine advance. It told you how to test whether the best strategy out of a large search actually beat the benchmark, or whether someone was always going to win by luck. It was rigorous, it was practical, and it was widely adopted.

It also had a flaw that, once pointed out, is impossible to unsee.

Under the Reality Check, you can make a real, genuinely profitable strategy fail the test simply by adding a pile of obviously terrible strategies to your search.

That is absurd. Adding junk to your candidate list should not destroy your ability to detect the gold that was already there. The junk carries no information about the gold. In 2005, Peter Reinhard Hansen diagnosed exactly why this happens and built a test that does not do it.

The problem: irrelevant alternatives poison the benchmark

To see the flaw, you have to remember how the Reality Check works. It asks: if none of my strategies had any skill, how good would the best one look by luck alone? It builds that "no skill" world by bootstrapping, runs all the candidate strategies through it, records the best score each time, and compares your real winner against that distribution.

Here is the trouble. To build the "no skill" world, White's procedure has to assume something about how good the strategies are under the null hypothesis. The Reality Check takes the most cautious possible assumption: it treats every strategy as if it were sitting exactly at the boundary, precisely as good as the benchmark and no better.

That is a safe assumption in the sense that it never gives false positives. But it is wildly pessimistic when applied to a strategy that is, in reality, awful. A strategy that loses money hand over fist is being credited, in the simulation, with benchmark-level performance. And a horrible strategy is typically also a volatile one, so in the artificial worlds it can occasionally post a huge lucky score.

The consequence: your bad strategies inflate the "best score by luck" distribution. They push the bar upward. Your genuinely good strategy now has to clear a bar that was raised by the noise of strategies that never had a chance. Add enough of them and you can bury anything.

The key idea via analogy: the hopeless competitors should not set the record

Imagine you are trying to decide whether a sprinter is world class. You compare their time against the best time you would expect from a field of amateurs having a lucky day.

The Reality Check's mistake is to assume that every amateur in the field is secretly capable of world-class times, and then simulate what happens when one of them gets lucky. If you have a thousand amateurs and you pretend they are all latent Olympians, the "best lucky amateur" time you simulate will be absurdly fast, and your real sprinter will look ordinary next to it.

Hansen's fix is common sense, made rigorous. It has two parts.

First, judge each competitor on their own scale. Rather than comparing raw performance, Hansen uses a studentized statistic: he divides each strategy's outperformance by its own variability. This is the same instinct behind the Sharpe ratio. Beating the benchmark by two percent means something very different for a calm strategy than for a wild one. Studentizing puts every candidate on comparable footing and stops erratic strategies from dominating the simulation just because they are erratic.

Second, and this is the heart of the paper, look at the data and notice which competitors are hopeless. Hansen's test examines each candidate's actual track record. Strategies whose performance is so poor that they are very unlikely to have any real skill get effectively excluded from the null distribution. They are not allowed to set the bar. In statistical language, he replaces the worst-case assumption about the null hypothesis with a sample-dependent one: the data tells you where each strategy plausibly sits, and the simulation respects that.

The result is a test that is far more powerful, meaning far more likely to correctly detect a real edge when one exists, while still keeping the false positive rate under control. And crucially, it is robust to irrelevant alternatives: throwing extra bad strategies into your search no longer sabotages the test.

Why it mattered

• It made data-snooping tests usable in practice. Real strategy searches contain hundreds or thousands of candidates, and most of them are duds. That is the nature of research. A test that collapses in exactly that situation is a test nobody can use. Hansen's SPA test is the one people actually reach for.
• It exposed a subtle and general lesson about composite hypotheses. The Reality Check's problem, assuming the worst case for every competitor, is a pattern that shows up all over statistics. Hansen's remedy, letting the data inform the null, is a technique with reach far beyond finance.
• It became infrastructure. The SPA test, and its relatives like the Model Confidence Set that Hansen later developed with co-authors, are now standard equipment for comparing forecasting models in economics, volatility modelling, and quantitative trading.
• It quietly rehabilitated some strategies. Results that had been declared insignificant under the Reality Check were not always insignificant. Some of them had simply been drowned by the junk in their own candidate pool.

The honest limitations

• It does not solve the real problem, which is you. Like the Reality Check, the SPA test corrects only for the search you declare. Every idea you tried and abandoned before assembling your candidate list is invisible to it, and that hidden search is usually the biggest source of overfitting. No statistic can audit a researcher's memory.
• The threshold for "hopeless" is a judgment call. Deciding which strategies are poor enough to be excluded from the null involves a tuning parameter. Set it one way and you get a test close to White's. Set it another and you get a more aggressive one. The results are not entirely free of this choice.
• It still depends on the bootstrap being right. The whole thing rests on resampling the data in a way that preserves its dependence structure. Get the block length or the resampling scheme wrong and the artificial worlds are wrong, and no amount of clever null-hypothesis handling will save you.
• More power means more exposure to everything else. A more sensitive test detects real edges more often, but it also detects, more often, the artifacts of a subtly biased backtest. The SPA test cannot distinguish "genuine skill" from "look-ahead bias in the data pipeline." It only knows what you feed it.
• Statistical significance is not economic significance. Passing the SPA test tells you an edge is unlikely to be luck. It does not tell you the edge survives transaction costs, or that it is large enough to be worth trading.

The one-line takeaway

Hansen showed that White's Reality Check had a perverse flaw, adding useless strategies to your search made it harder to detect a genuinely good one, and fixed it by judging each strategy on its own volatility and refusing to let the hopeless candidates set the bar.