Most Published Factors Are Probably False: Harvey, Liu and Zhu Raise the Bar

The title of this paper begins with an ellipsis. That is not a typo. It is a joke, and it is the sharpest joke in modern finance.

For decades, the way to get published in a top finance journal was to write a paper called something like "Corporate Governance and the Cross-Section of Expected Returns," or "Advertising Spend and the Cross-Section of Expected Returns," or "Weather and the Cross-Section of Expected Returns." You find a company characteristic, you show that stocks with more of it earn higher returns, you clear the standard statistical bar, and you have a publication.

Campbell Harvey, Yan Liu and Heqing Zhu titled their paper with the blank space where your favourite characteristic goes, and then spent sixty pages explaining why most of those papers are probably wrong.

The problem: everyone was using a test designed for a single experiment

Here is the standard bar. You compute a t-statistic, which is roughly "how many standard errors away from zero is the effect I found?" If it clears about 2.0, you declare the result statistically significant. This corresponds to roughly a one in twenty chance that you would see an effect this big if the truth were zero.

That bar is perfectly sensible if you are running one test. One hypothesis, one experiment, one answer.

But finance is not running one test. Finance is running a factory. Harvey, Liu and Zhu went and counted: they documented hundreds of factors claimed in the published literature to predict returns, and the count was still climbing steeply. And the published papers are just the survivors. For every factor that made it into a journal, an unknown number were tested, found wanting, and quietly abandoned. Journals do not publish "we looked at rainfall and it does not predict returns."

So the real number of tests is not hundreds. It is unknown, and much larger.

And here is the arithmetic that should ruin your day. If a one in twenty false positive rate is your standard, and the profession runs a thousand tests on characteristics that genuinely have no predictive power, then roughly fifty of them will clear the bar by pure chance. They will be published. They will be given plausible economic stories after the fact, because a good storyteller can rationalize any pattern. And they will be completely fake.

The key idea via analogy: the bar has to rise with the crowd

Suppose one person in a room claims they can predict a coin flip, and then does it correctly six times running. That is impressive. The odds of that by luck are about one in sixty-four.

Now suppose there are five hundred people in the room, all guessing, and you only pay attention to whoever happened to guess right six times running. You should expect several such people to exist. Their success tells you nothing at all. The evidential value of a result depends on how many people were trying.

This is called the multiple testing problem, and statisticians have known about it for a very long time. Genomics researchers, who scan tens of thousands of genes at once, had already been forced to confront it. Finance, oddly, had not.

Harvey, Liu and Zhu's contribution was to import that statistical machinery into asset pricing and work out what the honest bar should be. They applied several standard multiple-testing corrections, some that control the chance of making any false discovery, and some that control the fraction of discoveries that are false, and made a careful attempt to estimate how many unpublished tests were lurking behind the published ones.

Their conclusion, and this is the number the profession remembers: a t-statistic of 2.0 is nowhere near good enough. A newly proposed factor should be required to clear a t-statistic of about 3.0.

That sounds like a small change. It is not. Moving from 2.0 to 3.0 is roughly the difference between "one in twenty" and "one in a thousand." It is a demand for dramatically stronger evidence. And it means, by the authors' own reckoning, that a large share of the previously published factor literature would not survive the corrected test.

Why it mattered

• It is the paper that named the "factor zoo." The image of hundreds of jostling, dubious, half-tame factors stuck instantly, and it changed how people talk about the whole enterprise.
• It changed editorial standards. Top journals began, in practice, to demand more from new factor papers: higher significance, out-of-sample evidence, and an economic mechanism rather than a story invented afterwards. The bar genuinely moved.
• It gave practitioners a filter. If you are a quant deciding which of the published anomalies to actually trade, this paper hands you a triage rule. Anything that barely cleared the old bar should be treated as unproven. Only the factors with really strong statistical evidence and a credible economic reason deserve your capital.
• It reframed the replication crisis in finance. Psychology and medicine had already had their public reckonings with false positives. This paper was finance's. It set up the later replication studies that tried to re-run the published anomalies from scratch, many of which failed.

The honest limitations

• The number of tests is fundamentally unknowable. The entire correction hinges on estimating how many hypotheses have really been tried, including all the ones that died unpublished in a thousand PhD students' laptops. Harvey, Liu and Zhu make a serious, careful attempt at this, but it is an estimate resting on assumptions. If the true number is much larger, even a t-statistic of 3.0 is too generous.
• A higher bar throws away true discoveries too. Statistics has an unavoidable trade: making it harder to accept false factors also makes it harder to accept real ones, especially real ones with modest effects. There is no setting of the dial that gives you only truth.
• Not all tests are independent, and the correction assumes a lot. Many published factors are variations on the same underlying idea (there are a dozen ways to measure "value"). Treating them as separate independent tests overstates the multiplicity. The authors are aware of this, but the adjustment is imperfect.
• It is a purely statistical critique. A factor can clear any t-statistic you like and still be a mirage produced by a data error, survivorship bias, or an economic regime that has ended. Statistical significance was never the only thing that could go wrong.
• The 3.0 threshold gets treated as gospel. It was a considered recommendation given a set of assumptions, not a law of nature. It is now quoted as if it were engraved somewhere, which is exactly the kind of uncritical acceptance the paper was written to fight.

The one-line takeaway

Harvey, Liu and Zhu pointed out that finance has been running thousands of statistical tests while using a significance bar designed for running one, and concluded that most of the hundreds of published "factors" are probably false discoveries, with the honest bar for a new one being a t-statistic of about 3.0 rather than 2.0.