Shiller's Excess Volatility: Stock Prices Move Far Too Much

In 1981, Robert Shiller published a paper with one of the great chart-based arguments in the history of economics. It contained a picture so simple, and so damaging to the prevailing view, that people are still arguing about it four decades later. Shiller would eventually share a Nobel Prize, notably alongside Eugene Fama, whose efficient markets view this paper attacked, which tells you something about how unresolved the fight remains.

The problem: what is a stock price supposed to be?

Standard theory has a clean answer. A share's price today is the market's best estimate of the value of all the cash it will ever pay you, discounted for time and risk. In the simplest version, that means a stock's price should equal the market's rational forecast of its future dividends.

That is a forecast, and forecasts have a very well known statistical property. A forecast should be less volatile than the thing it is forecasting. Your guess about tomorrow's temperature should wobble less than the temperature itself, because a good forecast averages out the noise you cannot predict. If your forecast swings around more wildly than reality does, you are not forecasting. You are just being erratic.

Shiller realized this gives you a hard, testable prediction about the stock market. And it is a prediction that requires no assumptions about how investors form expectations, only that they are rational.

The key idea, via analogy

Suppose a bookmaker is setting odds on a marathon runner's finishing time. Before the race, the bookmaker's estimate should move around a little as news arrives: an injury report, the weather forecast. But the estimate should be calmer than the range of actual finishing times, because the bookmaker is trying to average across the possibilities.

Now imagine you watch the bookmaker's estimate swing from two hours to four hours and back, several times, while the eventual outcomes of similar races cluster tightly around three hours. You would conclude the bookmaker is not processing information. Something else is going on.

Shiller ran exactly this comparison for the US stock market, using data going back to the 19th century. Here is the ingenious move. Because he was looking at historical data, he could compute the thing prices were supposedly forecasting. He constructed what he called the ex post rational price: for every year in the past, work out the dividends that actually got paid afterwards, discount them back, and ask what the price should have been if investors had had perfect foresight.

Then he plotted the two lines on the same chart.

The perfect-foresight line is remarkably smooth. It drifts gently upward over the decades, because real dividends, in aggregate, are a fairly stable, slowly-growing stream. Companies work hard to smooth their dividends and rarely slash them wildly.

The actual price line is a rollercoaster. It swings violently, repeatedly, in booms and crashes, wandering far above and far below the smooth line it is supposedly tracking.

The forecast is many times more volatile than the thing being forecast. That is not supposed to be possible. Shiller's punchline: stock prices move too much to be explained by news about future dividends.

Why it mattered

• It was the first hard statistical strike against the efficient markets hypothesis. Earlier attacks were about whether you could beat the market, which is easy to dismiss. Shiller's test was different: it did not ask whether anyone could profit, it asked whether prices were even internally consistent with the theory. And they were not.
• It launched behavioral finance as a serious empirical discipline. If prices are not moving on dividend news, what are they moving on? Shiller's answer, developed over subsequent decades, was fashions, herding, waves of optimism and pessimism. The excess volatility finding is behavioral finance's founding empirical fact.
• It reframed the question of what discount rates are. The efficient markets response, which is entirely legitimate, is that prices can also swing because the discount rate changes, meaning the return investors demand rises and falls over time. Shiller's test essentially proves that something other than dividend news drives prices, and forced the profession to take time-varying discount rates seriously. That, ironically, became one of the most productive lines in rational finance.
• It made the CAPE ratio famous. The practical descendant of this work is Shiller's cyclically adjusted price-to-earnings ratio, which is still one of the most-watched long-run valuation gauges in the world.

The honest limitations

This paper was attacked hard, and some of the attacks landed.

• Discount rates might just vary a lot. The original test assumed a constant discount rate. If the return investors demand rises in recessions and falls in booms, prices will swing far more than dividends without any irrationality at all. This is the standard rational rebuttal, and it is not a dodge; Campbell and Shiller's own later work builds a framework that accommodates it explicitly.
• The statistics were contested. Marsh and Merton, among others, argued that the volatility bounds were biased by the way the tests handled non-stationary, trending data and by the fact that firms deliberately smooth dividends. If managers set dividends by smoothing, the "smooth dividend line" is partly an artifact of corporate policy, not proof that fundamentals are smooth.
• Terminal value assumptions matter. Computing the perfect-foresight price requires guessing a value for the stock at the end of the sample, and the results depend on that guess more than one would like.
• Dividends are not everything. Companies increasingly return cash via buybacks rather than dividends, so a dividend-only measure of fundamentals is incomplete for the modern era.

The one-line takeaway

Shiller showed that stock prices swing far more violently than the smooth stream of dividends they are supposedly forecasting, and since a forecast should never be more volatile than the thing it forecasts, something other than rational news about fundamentals is moving the market.