Your Risk Model Was Right for the Wrong Reason: Christoffersen on Interval Forecasts

A risk manager says: "I am 95% confident that tomorrow's loss will not exceed 2 million." That is an interval forecast, and it is the fundamental output of every risk system in the world. Value at Risk is exactly this.

Now, a year later, how do you check whether the risk system was any good?

The obvious answer, the one everyone used, is to count. Out of 250 trading days, the loss should have exceeded the limit about 5% of the time, so roughly 12 or 13 times. Count the breaches. If you got 12, congratulations, your model is fine.

Peter Christoffersen showed that this test is dangerously incomplete, and that a model can pass it while being spectacularly, obviously broken.

The problem: getting the count right tells you almost nothing

Consider two risk models. Both produce a 95% daily loss limit. Over one year, both are breached exactly 13 times, which is precisely what a correct model should do.

Model A's breaches are scattered randomly through the year. One in February, one in April, a couple over the summer, and so on. No pattern.

Model B's breaches all happen in the same two weeks in October. Thirteen breaches, back to back, during a market crash. For the rest of the year, not one.

Both models pass the counting test with flying colours. Model B is catastrophically bad, and any risk manager would rather be shot than use it.

Why? Because Model B's failures cluster. Its limit is wrong exactly when it matters. During the crisis, the model was blind for two straight weeks while the firm bled. A risk system whose failures cluster is not measuring risk, it is measuring the calendar.

And this is not a hypothetical failure mode. It is precisely what happens when a risk model ignores volatility clustering. A model that uses a constant volatility estimate will be far too loose in calm periods (never breached) and far too tight in turbulent ones (breached every day). It will get the average count right and be wrong every single day.

The key idea via analogy: the weather forecaster who says "no rain" every day

Suppose you live somewhere it rains 5% of the days. A forecaster who says "no rain" every single day will be right 95% of the time. His hit rate is perfect. He is also completely useless, because his forecast contains no information: he is never right when it matters.

The counting test is checking his hit rate. Christoffersen's contribution is to check something more demanding: is he right for the right reasons?

The solution: split the test in two

Christoffersen's framework is elegant because it decomposes a single vague question, "is my risk model good," into two sharp and separately testable ones.

Test one: unconditional coverage. Does the model breach at the right rate? Over the whole sample, is it violated about 5% of the time? This is the old counting test, made rigorous with a proper statistical test rather than eyeballing.

Test two: independence. Are the breaches independent of each other? Specifically, does a breach today make a breach tomorrow more likely? A correct model would say no: breaches should arrive like a coin landing heads, with no memory. If a breach today predicts a breach tomorrow, your model's failures are clustering, and it is systematically blind during turbulent periods.

Then, crucially, he combines them into a single conditional coverage test that demands both at once: the right number of breaches, and those breaches arriving unpredictably.

The insight underneath is what makes this a volatility paper as much as a risk paper. A risk model passes the independence test only if it correctly tracks time-varying volatility. If your volatility forecast rises before turbulent periods, your limit widens with it, and breaches stay scattered. If your volatility forecast is static, your breaches will pile up in the storms, and this test will catch you.

Christoffersen was explicit about this motivation. Most of the interval forecasting literature had been built on the assumption of constant error variance, which is exactly the assumption that finance data has been screaming against since Mandelbrot. His framework is designed for the world where higher-moment dynamics are present.

Why it mattered

• It became the standard backtest. Christoffersen's tests are implemented in essentially every commercial and academic risk package. If you have ever run a VaR backtest, you have almost certainly run his tests, probably without knowing whose they are.
• It entered regulation. Bank regulators require VaR models to be backtested, and the practice of examining not just breach counts but breach clustering is standard supervisory expectation.
• It made a sharp point about what risk models are for. A risk model is not supposed to be right on average. It is supposed to be right conditionally, day by day, updating as conditions change. The independence test is what enforces that distinction.
• It rewards good volatility modelling directly. It provides a clear, practical answer to "why should I bother with GARCH instead of a rolling standard deviation." Because the rolling standard deviation will fail this test, and GARCH will not.

The honest limitations

• The tests are underpowered. With 250 observations and a 5% limit, you expect about 12 breaches. Detecting subtle misspecification from 12 events is statistically hopeless. The tests reliably catch only badly broken models. A model that is somewhat wrong can sail through.
• The independence test only looks one step back. In its standard form it checks whether a breach today predicts a breach tomorrow. A model whose breaches cluster over longer, more diffuse periods can evade it.
• Passing is not proof. These are tests of a hypothesis. Failing to reject the model is not the same as the model being right, particularly when power is low.
• It says nothing about how bad the breaches are. This is the deepest limitation. A breach is a breach in this framework, whether you lost one dollar more than your limit or a hundred million more. Value at Risk, and therefore these tests, is blind to the size of the tail. That blindness is exactly what motivated the shift toward expected shortfall, which asks not "how often do I breach" but "how bad is it when I do."
• It assumes the limit is the model's honest output. In practice, VaR numbers are sometimes nudged for commercial reasons, and no statistical test can save you from that.

The one-line takeaway

Christoffersen showed that counting how often your risk limit is breached is not enough, because a model can breach the right number of times and have every single breach land in the same terrible week, and he built the test that demands breaches be not just correctly numbered but unpredictable, which is a demand only a model that genuinely tracks changing volatility can satisfy.