We Tried to Replicate 447 Anomalies. Most Failed.

Psychology had its replication crisis. Medicine had one. Finance, it turns out, had one too, and this is the paper that documented it.

Kewei Hou, Chen Xue and Lu Zhang did something that sounds boring and turned out to be devastating. They took 447 published stock return anomalies, every one they could find in the literature, and simply tried to rebuild them. Carefully. Consistently. With sensible methods.

Nearly two thirds of them did not survive.

The problem: microcaps were doing all the work

To understand why so many anomalies evaporated, you need to understand one specific methodological choice that had quietly corrupted a large part of the literature.

The US stock market contains a few thousand companies. A large number of them are microcaps: tiny firms, often worth less than a hundred million dollars, thinly traded, expensive to trade, and frequently impossible to short. Collectively they are a rounding error in terms of the market's total value. They are perhaps 3% of total market capitalisation. But they are the majority of the companies by count.

Now, how do you test whether a signal predicts returns? A very common approach was to run a regression across all stocks, or to form portfolios that weight every stock equally. Both of those approaches give a microcap the same influence as Apple.

The result is that a huge amount of published "evidence" was really a statement about tiny, untradeable companies. If a signal works beautifully in stocks nobody can buy and does nothing in stocks anyone can actually own, is that an anomaly? For an economist asking whether markets are efficient, arguably yes, a little. For an investor, absolutely not.

The key idea via analogy: measure it the way an investor would

Hou, Xue and Zhang re-tested everything using two disciplines:

1. Value-weighted returns. Weight each stock by its market value, so big companies count more, exactly as they would in any real portfolio. A signal that only works in microcaps will now show almost nothing, because the microcaps barely register.
2. NYSE breakpoints. When splitting stocks into groups (say, cheapest 10% by valuation), define the cut-offs using only the larger NYSE-listed companies. Otherwise the "top decile" ends up stuffed with hundreds of tiny stocks, and you are back to measuring the microcap tail.

Neither of these is exotic. Both are what you would do if you actually wanted to know whether a signal makes money. And with these disciplines applied consistently across 447 anomalies, the results were brutal:

• 286 of the 447 anomalies, about 64%, were statistically insignificant at the conventional 5% level.
• The trading frictions literature was almost entirely wiped out: 102 of 106 anomalies in that category, roughly 96%, failed. This is grimly logical. Anomalies based on liquidity, volume, and trading costs are, almost by construction, phenomena of small illiquid stocks. Once you stop letting microcaps dominate, there is nothing left.
• Even among the 161 anomalies that did survive, their magnitudes were typically much smaller than originally reported.
• And of those 161 survivors, the authors' own q-factor model left 115 of their alphas insignificant, meaning most of the real ones were not new phenomena at all, just repackaged exposure to investment and profitability.

Their conclusion is stated plainly: capital markets are more efficient than previously recognized. The mountain of published evidence against market efficiency turned out to be, in large part, a mountain of methodology.

Why it mattered

• It is finance's replication crisis, in one document. Not a theoretical worry about statistics, an actual attempt to rebuild the literature, with a scoreboard at the end.
• It changed publication standards permanently. Value-weighted results and NYSE breakpoints are now expected. A referee at a serious journal will ask for them, and many pre-2015 findings simply cannot supply them.
• It complements the multiple-testing critique. Harvey, Liu and Zhu argued statistically that most published factors must be false positives. Hou, Xue and Zhang went and checked, and found precisely that. Two completely different methods reaching the same conclusion is powerful.
• It is the most useful piece of advice a practitioner will ever get for free. Before you build a strategy from a paper, ask where the returns come from. If they come from the smallest 3% of the market, they are probably not yours to have.

The honest limitations

• "Insignificant" is not the same as "false." A real but modest effect can fail a significance test, particularly once value-weighting throws away most of the sample's statistical power. Some of the 286 casualties may be real effects that this method is not sensitive enough to detect. That is a genuine cost of the approach.
• Value-weighting is itself a choice. It answers the question "could a large investor have made money?" It does not answer "is the market efficient?", because a mispricing in small stocks is still a mispricing. The two questions are different, and the paper's method favours one.
• The authors are not neutral. They are the proprietors of the q-factor model, and the paper concludes that the q-factor model explains most of the survivors. That is a real finding, but it is worth noting who is scoring the contest.
• Replication choices are still choices. Rebuilding somebody else's anomaly requires dozens of decisions about data, timing and definitions. Original authors have contested some of those decisions, and a few of the disputes are unresolved.

The one-line takeaway

Hou, Xue and Zhang rebuilt 447 published anomalies with methods that stop tiny untradeable stocks from dominating the results, and about two thirds of them vanished, which suggests the literature's case against market efficiency was built substantially on a methodological artefact.