The More Data You Use, the Worse It Gets: A Tale of Two Time Scales

The theory of realized volatility makes a clean promise: the more finely you sample prices, the more accurate your volatility measurement becomes. Sample every five minutes and you get a decent estimate. Sample every minute and you get a better one. Sample every tick, using every single trade, and you should approach the exact truth.

Anyone who has actually tried this knows it is a disaster. As you sample more finely, your realized volatility estimate does not converge on the truth. It inflates, wildly and without limit. Use every tick on a liquid stock and you will get a volatility number several times larger than any sane estimate.

This is one of the great embarrassments of high-frequency econometrics: the theory says use all the data, and using all the data produces nonsense. In 2005, Lan Zhang, Per Mykland and Yacine Aït-Sahalia explained exactly why, and built the fix.

The problem: at fine timescales you are not measuring the price

The theory assumes you observe the efficient price, the true, model-free value of the asset, moving continuously. You do not. You observe transaction prices, and a transaction price is the efficient price plus a layer of junk:

• Trades happen at the bid or at the ask, so consecutive trades bounce back and forth across the spread even when nothing has changed. This alone creates fake movement.
• Prices are quoted in discrete ticks, so they are rounded.
• Orders arrive irregularly and in clumps.
• Large orders push prices temporarily and then let them spring back.

Collectively this junk is called market microstructure noise. At the five-minute scale it is a minor nuisance because the real price has had time to move a lot compared to the noise. But at the tick scale, the real price has barely moved at all while the noise is exactly as big as ever. The signal shrinks and the noise does not.

So when you sum up squared tick-by-tick returns, you are mostly summing up squared noise. And the more finely you sample, the more noise terms you add. Your estimate diverges. It does not converge to true volatility, it converges to a measure of how much the bid-ask bounce bounces.

This explains the field's uncomfortable compromise: everyone samples every five minutes, not because five minutes is theoretically special, but because it is a superstitious middle ground between "too little data" and "too much noise." The authors were not satisfied with a superstition.

The key idea via analogy: use the noise to cancel the noise

Here is the elegant move.

Suppose you measure something twice with two different instruments: one that is precise but heavily biased, and one that is imprecise but unbiased. Neither is any good alone. But if you can figure out how much the biased instrument is biased by, you can subtract the bias and keep the precision.

That is exactly what Zhang, Mykland and Aït-Sahalia do, and their genius is realising that the biased instrument tells you its own bias.

Time scale one: sample as fast as you possibly can, every tick. This estimate is hopelessly contaminated. But that is the point. Because it is so contaminated, it is essentially a pure measurement of the noise. It tells you how big the microstructure junk is.

Time scale two: sample slowly, on a sparse grid. This estimate has much less noise contamination, but it throws away most of the data, so it is imprecise. Crucially, the authors do this cleverly: rather than picking one sparse grid (say, prices at 9:30, 9:35, 9:40) and discarding the rest, they use every possible sparse grid (also 9:31, 9:36, 9:41, and 9:32, 9:37, 9:42, and so on) and average the results. This way no data is wasted, and the sparse estimate becomes much more precise.

Then combine. Take the averaged sparse estimate, and use the fast estimate to work out how much noise bias remains in it, and subtract it off. The noise, measured at the fastest scale, is used to correct the estimate built at the slow scale.

They call the result the two-scales realized volatility estimator, and they prove it does what realized volatility was always supposed to do: converge to the true integrated volatility as you get more data, even in the presence of noise.

Why it mattered

• It was the first consistent estimator under noise. That is the headline. Before this paper, every practical approach was a compromise. This was the first construction with a proof that more data genuinely means a better answer.
• It stopped the waste. The five-minute convention throws away well over 99% of the observations in a liquid name. This paper showed you do not have to.
• It opened a research programme. A stream of refined noise-robust estimators followed, including multi-scale versions, pre-averaging methods and realized kernels. Almost all of them cite this paper as the starting gun.
• It reframed noise as information. Microstructure noise is not just an obstacle. It is a measurable quantity that carries information about liquidity, spreads and trading frictions. Once you have to estimate it, you may as well look at it.

The honest limitations

• The noise assumptions are strong. The clean version of the theory assumes noise is independent from one observation to the next and independent of the true price. Reality is not so kind: the bid-ask bounce is serially correlated, and informed trading means the noise is correlated with the price move. Later work, including by the same authors, relaxes these assumptions, but at the cost of more complexity.
• It is fiddly to implement. Choosing the number of sparse grids, handling irregularly spaced trades, deciding whether to use trades or quotes, dealing with the opening auction. There are a lot of practical decisions, and the estimator is sensitive to some of them.
• The efficiency gain is real but bounded. The estimator converges more slowly than an oracle who could see the true price. Noise imposes a genuine, unavoidable statistical cost.
• It needs a liquid asset. On a stock that trades a hundred times a day, there is no "fastest time scale" worth speaking of, and the whole construction is moot.
• Five minutes is still everywhere. A slightly humbling coda: despite the theory, an enormous amount of applied work still uses simple five-minute realized volatility, because it is easy, robust, and usually good enough. The sophisticated estimators win on theory and often only modestly in practice.

The one-line takeaway

Zhang, Mykland and Aït-Sahalia explained why using every tick makes your volatility estimate worse rather than better, at fine timescales you are measuring the bid-ask bounce rather than the price, and then turned that vice into a virtue by using the tick-level data to measure the noise, and the noise measurement to clean up an estimate built on slower data.