A Century of Chart Patterns, Put on Trial: Sullivan, Timmermann and White

In 1992, a well-known study by Brock, Lakonishok and LeBaron tested a set of twenty-six simple technical trading rules, things like moving average crossovers and trading range breakouts, on ninety years of Dow Jones data. Several of them looked profitable. The finding caused a stir, because it seemed to be evidence that the chart-reading crowd, long dismissed by academics as astrologers, might have been onto something.

Seven years later, Ryan Sullivan, Allan Timmermann and Halbert White came back with an uncomfortable question: where did those twenty-six rules come from?

Because nobody wakes up one morning and invents exactly twenty-six trading rules from first principles. Those rules were the survivors of a century of collective tinkering by traders and analysts. The parameters, the twenty-day window rather than the seventeen-day window, the one percent band rather than the half-percent band, were not chosen at random. They were chosen because, at some point, somebody noticed they worked on this very data.

That is data snooping, and it does not stop being data snooping just because the snooping was done by other people, over decades, before you were born.

The problem: you inherited a search you never ran

This is the paper's deep and unsettling insight, and it applies far beyond technical analysis.

When you test a rule that the world has already been staring at for a hundred years, you are not running a fresh experiment. You are examining the winner of an enormous, undocumented tournament. The tournament happened. You just did not attend. Every trading rule that failed has been quietly forgotten, and the ones that survived into the textbooks survived precisely because they looked good on the historical record you are now using to test them.

Testing such a rule with a standard significance test is like meeting the last person standing in a coin-flipping contest and concluding, from their remarkable streak, that they have a gift.

So Sullivan, Timmermann and White set out to reconstruct the tournament.

The key idea via analogy: put the whole field back on the track

Their method has a simple, brutal logic: if the rule you are testing was picked from a large universe, then you must test it against that entire universe, not on its own.

So they built the universe. Rather than taking twenty-six rules on faith, they generated the full space from which such rules plausibly come: every reasonable variation of moving average rules, filter rules, support and resistance rules, channel breakouts, and on-balance volume rules, across a huge grid of parameter choices. The result was a universe of nearly eight thousand trading rules.

Then they applied Halbert White's own Reality Check. The procedure, in plain terms:

1. Run all eight thousand rules on a century of daily Dow data and record the best performer.
2. Use a bootstrap to build thousands of artificial price histories that have the same statistical texture as the real one but contain no genuine profitable rules by construction.
3. In each artificial world, run all eight thousand rules again and record the best score.
4. You now have a distribution of "how good does the best of eight thousand rules look when none of them work?"
5. Ask whether the real champion beats those fake champions.

This is the crucial move. The benchmark is not "did the best rule beat buy-and-hold?" It is "did the best rule beat what the best of eight thousand worthless rules would have achieved by luck?" That bar is far, far higher, and it is the only honest one.

The analogy: it is not enough to know that the winner of the race ran fast. You have to know how many people were in the race, and how fast the fastest of them would run if they were all mediocre.

The finding, in two halves

The results are more nuanced than either camp wanted, which is usually a sign of honest work.

On the historical sample that Brock, Lakonishok and LeBaron had studied, the best rule survived. Even after correcting for the full eight thousand candidates, the top performer still looked genuinely superior. The technical analysts were not obviously wrong about the past.

But then the authors did the thing that really counts: they took the winning rule forward, into the years that came after the original study, and traded it. And in that subsequent out-of-sample period, the champion did not deliver. Its superior performance did not carry over.

That two-part result is the whole story of quantitative research in miniature. A pattern can be real in the historical data, statistically robust even after a serious multiple-testing correction, and still be worthless going forward. Markets change. Patterns that were once exploitable get arbitraged away once enough people know about them. The very act of publishing a profitable rule is often the act that kills it.

Why it mattered

• It is the definitive empirical demolition of naive technical-rule backtesting. Not because it proved technical analysis never works, but because it proved that the standard way of testing it was hopelessly inadequate.
• It made "the universe of rules" a required concept. After this paper, any serious backtest has to answer: what is the full space of strategies I could have selected from, and have I accounted for it? That question is now built into how professional quants think.
• It showed that inherited hypotheses are contaminated hypotheses. This is the underrated lesson. Even if you only tested one rule, if the profession tested ten thousand before handing it to you, you are the beneficiary of a search you never ran and must pay for anyway.
• It demonstrated the Reality Check on a problem people cared about. White's method could have remained a technical curiosity in an econometrics journal. This paper put it to work on a question with real stakes, in the Journal of Finance, and made it famous.

The honest limitations

• The universe of eight thousand rules is still a choice. The authors did an admirable job of enumerating the plausible rule space, but "plausible" was their judgment. A different researcher would build a different universe, and the correction depends on it. There is no objectively correct denominator.
• It only covers simple, mechanical rules. The universe is built from moving averages, filters, and breakouts. A modern quant strategy that blends dozens of features with a machine learning model is not in that space, and this paper says nothing directly about it, though the logic applies with equal force.
• Transaction costs are a persistent headache. Many technical rules trade frequently. Whether they survive realistic costs, across a century in which those costs changed enormously, is a hard question that any single answer will oversimplify.
• A century of one index is a lot of data and also very little. It is a long time series, but it is one market, and the number of genuinely independent market regimes inside it is much smaller than the number of days.
• The negative out-of-sample result does not prove the rule was fake. It could have been real and then decayed, which is a different story with different implications. Distinguishing "never existed" from "existed and died" is one of the hardest problems in the field, and this paper cannot settle it.

The one-line takeaway

Sullivan, Timmermann and White tested nearly eight thousand technical trading rules on a century of Dow data and showed that you must judge the winner against the whole universe it was drawn from, and that even a rule which survives that test may quietly stop working the moment you try to trade it forward.