Don't Trust Your Return Forecasts: Jorion's Bayes-Stein Shrinkage

Ask someone to forecast a stock's expected return and their first instinct is to look at what it has averaged historically. This is the single most natural thing to do, and it is almost useless.

Here is why, and it is a fact that surprises people every time. Suppose a stock has an average annual return of 8 percent and a volatility of 20 percent. To estimate that 8 percent average with any reasonable precision from the data, you need an enormous amount of history. Not a few years. Decades, and often centuries. Volatility can be measured well from a few years of daily data, because more frequent observation genuinely helps you pin it down. Expected return cannot: sampling more often does not help, only a longer calendar span does, and a longer span means the world has changed underneath you.

So the input that a mean-variance optimizer is most sensitive to is the one you can estimate least well. Philippe Jorion's 1986 paper is one of the earliest and most influential attempts to do something about it.

The problem: the optimizer takes your worst numbers most seriously

Chopra and Ziemba would later quantify it: errors in expected returns are roughly ten times more damaging to portfolio choice than errors in variances, and about twenty times more damaging than errors in covariances. Michaud named the mechanism: the optimizer preferentially loads up on whichever asset your noisy estimate has flattered most.

Given the sample average return of ten different assets over, say, ten years, the highest of those ten averages is very likely high partly because that asset got lucky. Feed those averages into an optimizer and it will hunt down the luckiest one and put a large chunk of your money into it, precisely at the moment when its luck is due to run out.

The classical statistician's response is that the sample mean is unbiased, so it is the right estimator. Jorion's response, following Charles Stein's famous result in statistics, is that unbiased is the wrong thing to want.

The key idea via analogy: rookie batting averages, again

You are managing a baseball team and you want to project each player's true ability from twenty at-bats each. One rookie hit .400. Another hit .150.

Nobody sane projects .400 and .150 going forward. You drag both estimates toward the league average, because you know twenty at-bats is not enough to separate skill from luck. The rookie who hit .400 was probably good and lucky. The one who hit .150 was probably bad and unlucky. Pull both toward the middle.

That is shrinkage, and Stein's remarkable theorem says something stronger than "this feels sensible." It says that when you are estimating several things at once, shrinking all of your individual estimates toward a common center produces a provably better overall result than using each individual sample average, even though every individual shrunk estimate is now biased. You accept bias, you get a bigger reduction in noise, and you come out ahead. This felt so wrong when Stein proved it that it was called a paradox.

Jorion applied exactly this to expected returns.

What Jorion actually did

The Bayes-Stein estimator he built has three ingredients.

The individual estimates: each asset's historical average return. Noisy, and prone to being extreme in whichever direction luck went.

A shrinkage target: a common anchor for all assets. Jorion's choice was the expected return of the global minimum-variance portfolio, a sensible, data-driven grand mean. It represents a kind of neutral "no strong view" prior on what any asset earns.

A shrinkage intensity determined by the data. How hard should you pull toward the anchor? Jorion derived it, and it behaves exactly as you would want: if you have very little data, or your assets look very noisy, shrink hard. If you have a lot of data and the sample averages are tightly determined, shrink less. This is the empirical Bayes idea, using the data itself to tell you how much to trust the data.

The result is a set of expected returns that are compressed together. The extreme winners are pulled down, the extreme losers are pulled up, and the optimizer no longer has wildly divergent numbers to fall in love with. Naturally, the portfolios that come out are much less concentrated and much more stable over time.

And they perform better. In out-of-sample tests, portfolios built on Bayes-Stein expected returns outperformed those built on the raw sample means, which is the only test that matters.

Why it mattered

• It attacked the biggest problem, not the convenient one. Ledoit and Wolf's more famous shrinkage work targets the covariance matrix. Jorion went after expected returns, which is where the real damage is. It is a harder problem and a bigger payoff.
• It brought Stein's paradox into finance. The general lesson, that when estimating many things simultaneously you should deliberately bias your estimates toward each other, is one of the deepest ideas in statistics. Jorion is the reason it is now standard practice in portfolio construction.
• It is a direct ancestor of Black-Litterman. Black and Litterman's model shrinks an investor's views toward the market's implied equilibrium returns. That is precisely the same idea with a different anchor: a defensible prior, and an investor's noisy views pulled toward it with an intensity that reflects confidence.
• It is the same principle as modern regularization. Ridge regression, LASSO, and the whole apparatus of penalized estimation in machine learning are all doing what Jorion did: accept bias to buy a bigger reduction in variance. Anyone who has tuned a regularization parameter has been doing Jorion's job without knowing it.

The honest limitations

• The anchor matters enormously. Shrinking toward a bad target simply moves your estimates toward a different wrong answer. The minimum-variance portfolio's expected return is a reasonable choice, but it is a choice, and it imports its own assumptions.
• It compresses information you might genuinely have. Shrinkage is agnostic. It pulls in your good signals along with your noise. If you actually possess real, hard-won information that one asset will outperform, Bayes-Stein will dutifully shrink that view toward the mean along with everything else. It cannot tell insight from luck, which is exactly why it is safe, and exactly why it is limiting.
• It softens the problem, it does not solve it. Expected returns remain nearly unforecastable. Shrinkage stops the optimizer from behaving insanely on the basis of noise. It does not conjure predictive power that is not there.
• It assumes returns are drawn from a stable, well-behaved distribution. The derivation rests on statistical assumptions about the return process, and real returns are fat-tailed, time-varying, and regime-dependent.
• Even with shrunk means, mean-variance is fragile. Later work, notably DeMiguel, Garlappi and Uppal, found that Bayes-Stein portfolios still failed to consistently beat naive equal weighting out of sample. That is a sobering result: even the best-known fix for the biggest input problem was not enough to clear a very low bar.

The one-line takeaway

Jorion showed that historical average returns are so noisy that using them directly poisons a portfolio optimizer, and that deliberately dragging all of your return estimates toward a common anchor, making each one individually more biased, produces better portfolios in the real world, which is Stein's paradox arriving in finance.