How Risky Is Your Risk Number? Jorion on the Error Bars Around VaR

In the mid 1990s, Value-at-Risk swept through banking. JP Morgan's RiskMetrics had made it accessible, regulators were building capital rules around it, and every trading desk had a VaR number. The appeal was that it collapsed a bank's entire risk into one dollar figure, and one dollar figure is something a board of directors can actually discuss.

Philippe Jorion, who literally wrote the standard textbook on VaR, published a paper in 1996 that pointed at an embarrassing hole in this picture. Everyone was treating the VaR number as a fact. It is not a fact. It is an estimate, computed from a limited sample of data, and like every estimate it comes with sampling error.

The title says it all. Risk squared: the risk in your risk measure.

The problem: a number reported without error bars is a number pretending to be certain

Think about how a VaR number is produced. You take some history of returns, you estimate volatility and correlations from it, or you take the empirical distribution of past outcomes directly, and out pops a quantile: the loss you should exceed only 1 percent of the time.

Every step of that involved estimating something from a finite sample. So the resulting VaR is a random variable. Run the same procedure on a different sample from the same underlying world and you would get a different VaR.

How different? That is the question, and until this paper nobody had systematically answered it in a way risk managers could use.

The stakes are practical, not philosophical. If a bank reports a VaR of 10 million and the honest confidence interval around it runs from 6 million to 15 million, then the difference between that bank and another one reporting 12 million is meaningless noise. Capital rules, risk limits, and internal comparisons all silently assume a precision that is not there.

The key idea via analogy: the poll with no margin of error

Imagine a political poll that reports "candidate A has 52 percent support" and never mentions a margin of error. You would immediately object: with how many respondents? Plus or minus what? A 52 percent result from a sample of 100 people means nothing at all. From a sample of 50,000, it means something.

VaR was being reported like that poll: a point estimate with no margin of error attached. Jorion's paper is essentially the argument that risk reports need margins of error too, together with the statistical machinery to compute them.

And the news, when you do compute them, is uncomfortable in a specific and instructive way.

The problem gets worse the deeper into the tail you go. This is the crucial insight. To estimate the ordinary volatility of a portfolio, you can use all of your data: every observation tells you something about how much the portfolio typically moves. But to estimate a 99th percentile loss directly from the data, you are relying on the handful of observations that actually sit out in that tail. With 250 trading days of history, the 99 percent quantile is being pinned down by roughly two or three data points.

Two or three data points. That is the empirical foundation under a number that determines a bank's regulatory capital.

So there is a nasty trade-off baked into VaR:

• Choose a high confidence level (99 percent, 99.9 percent) to be prudent, and your estimate becomes wildly imprecise, because you are estimating a quantile from almost no observations.
• Choose a lower confidence level (95 percent) for statistical stability, and you are measuring a loss that is not really a tail event at all, which is not what a risk manager wants to know.

You cannot have both precision and extremity from a finite sample. That tension is intrinsic, and Jorion made it explicit.

What he showed

The paper lays out the statistical methodology for quantifying estimation error in VaR, and draws out the practical consequences.

Parametric methods are more precise, if you can stomach the assumption. If you are willing to assume a distribution, typically the normal, then you estimate VaR by estimating the volatility and scaling it, and volatility can be estimated from all your data, not just the tail. That gives a much tighter confidence interval. The price is that you have bet the whole thing on the distributional assumption, and financial returns are famously fatter-tailed than normal. You have traded sampling error for model error.

Non-parametric (historical simulation) methods make no distributional assumption but pay for it with enormous sampling error in the tail, for the reason above.

Longer samples help, but slowly, and they bring their own problem. Precision improves with the square root of sample size, so quadrupling your history halves your error. But a longer history reaches further back into a world that may no longer resemble the present one. You are trading sampling error for staleness.

The general lesson is that there is no free lunch in tail estimation. Every route to a more precise VaR involves an assumption that could be wrong, and every route that avoids assumptions leaves you with a number you cannot pin down.

Why it mattered

• It punctured the illusion of precision. The single most dangerous property of a VaR report is that it looks authoritative. A number with a dollar sign and no error bar invites false confidence. This paper made the error bars visible, and made ignoring them a choice rather than an oversight.
• It gave risk managers a real tool. Confidence intervals for VaR are something you can actually compute and report, and Jorion showed how. It also gives you a principled way to compare methods: which estimation approach gives me the tightest honest interval for my situation?
• It clarified the design trade-offs. The choice of confidence level, sample length, and parametric versus historical method are not arbitrary conventions. Each is a specific point on a precision-versus-assumption trade-off, and this paper is why practitioners can articulate that.
• It contributed to the broader case against naive VaR. Combined with Artzner's demonstration that VaR is not a coherent risk measure, and with the crises that followed, this line of work moved the profession toward expected shortfall and toward stress testing as complements to a single quantile number.

The honest limitations

• Estimation error is not the only error, and probably not the worst one. The paper quantifies the uncertainty within a model. It does not address the possibility that the model itself is wrong, that the distribution has shifted, or that the risk you are about to suffer is of a kind your data has never contained. Model risk dwarfs sampling risk in a crisis, and no confidence interval catches it.
• Confidence intervals for VaR themselves rest on assumptions. The machinery for computing the error bars needs its own statistical assumptions, typically about independence and stationarity of returns. Financial returns violate both, particularly through volatility clustering.
• Knowing your VaR is imprecise does not tell you what to do about it. A risk manager who learns that their 10 million VaR could plausibly be 15 million faces an awkward question: now what? Reporting a range is more honest, but capital rules and risk limits want a number.
• The deeper problem is VaR itself. All the precision in the world does not fix the fact that VaR ignores what happens beyond the threshold and can penalize diversification. A precisely estimated bad measure is still a bad measure.

The one-line takeaway

Jorion pointed out that Value-at-Risk is not a measurement but an estimate with a confidence interval around it, that the interval gets alarmingly wide precisely as you push toward the extreme tails you actually care about, and that every method for narrowing it does so by importing an assumption that could be wrong.