Is My Forecast Actually Better Than Yours? Diebold and Mariano

You have built two forecasting models. Over the last two years, Model A had a mean squared error of 4.1. Model B had 4.4. Model A wins.

Does it? Or did Model A just get lucky on a handful of days?

This is one of the most consequential questions in all of quantitative work, because the entire model-selection process, and therefore what you eventually put real money behind, hinges on it. And before 1995, the standard answer was essentially "look at the two numbers and see which is smaller." No standard error. No confidence interval. No way to distinguish a real improvement from noise.

Francis Diebold and Roberto Mariano fixed this, and their test is now the standard way to ask whether one forecast is genuinely better than another.

The problem: comparing forecasts is not like comparing anything else

You might think this is easy. Just do a t-test on the errors. But forecast errors are hostile to the usual assumptions in three distinct ways, and every one of them matters.

First, forecast errors are correlated over time. If you are making forecasts three months ahead, and you make a new one every month, then consecutive forecasts overlap. They cover overlapping windows of the future, so their errors are mechanically related. Feeding correlated errors into a standard t-test produces standard errors that are far too small, and confidence that is far too high.

Second, the two models' errors are correlated with each other. Both models are looking at the same world. When something unpredictable happens, both get it wrong at the same time. Any comparison that treats the two error sequences as independent throws away this shared shock and badly misjudges the uncertainty.

Third, forecast errors are not normally distributed. They are fat-tailed, skewed, and frequently have a non-zero average. Financial forecast errors in particular are as far from a tidy bell curve as it is possible to get.

Any one of these breaks the naive test. All three together make it worthless.

The key idea via analogy: race them head to head, not lap by lap

Here is Diebold and Mariano's move, and it is beautifully simple.

Stop trying to compare two distributions of errors. Instead, for each single day, compute one number: how much worse was Model B than Model A on this particular day? That is your loss differential. If it is positive, A won that day. If negative, B won.

Now you have collapsed two complicated, correlated error series into a single series of daily margins of victory. And the question you actually care about becomes trivially simple:

Is the average of that margin series significantly different from zero?

This is like comparing two runners. You could try to model each runner's speed distribution separately, accounting for the fact that they both run slower on windy days. Or you could just race them side by side repeatedly and record who won each race, and by how much. Racing them together automatically cancels out the wind. Whatever affected them both is already differenced away.

That is exactly what the loss-differential trick achieves. The common shock that fooled both models on a given day appears in both error terms and largely cancels when you subtract. What is left is the genuine relative performance.

The remaining problem is that the daily margins are still correlated over time. So Diebold and Mariano compute the average margin's standard error in a way that is robust to that serial correlation, using the same kind of correction Newey and West developed. The result is a single statistic that you can compare to a normal distribution.

The generality is the point

The reason this test spread everywhere is not just that it works. It is that it barely assumes anything.

• The loss function can be whatever you want. The test does not care whether you measure error as squared error, absolute error, or something wildly asymmetric. That last part is a big deal in finance. If under-predicting risk costs you your job and over-predicting it costs you a bit of return, then your loss function is not symmetric, and you should be selecting models on the loss that actually hurts you, not the one that is mathematically convenient. Diebold-Mariano lets you do that.
• The errors can be fat-tailed, biased, and correlated. All the pathologies that break the naive test are explicitly permitted.
• You need nothing but the two error series. You do not need to know how the models were built. You do not need them to be nested, or related, or even sensible. Two black boxes and their track records are enough.

Why it mattered

• It made "my model beats yours" a testable claim. Model comparison went from an informal eyeballing exercise to a statistical one, with an actual p-value attached.
• It became the standard in forecast evaluation. In economics, in weather, in energy, in finance, the phrase "we test for significant differences in predictive accuracy using Diebold-Mariano" appears in an enormous number of papers.
• It is directly relevant to every backtest. Your new strategy has a Sharpe of 1.3. The old one has 1.1. Is that an improvement or a coin flip? The Diebold-Mariano logic (difference the two series, test whether the mean of the difference is distinguishable from zero, with serially-robust errors) is exactly the right way to think about it, and is far more honest than comparing two numbers and declaring victory.
• It legitimised custom loss functions. It gave researchers permission to evaluate models on the criterion they actually care about rather than on squared error by default.

The honest limitations

• It was not designed for nested models, and it is misleading when used on them. This is the most important caveat and the most frequently violated. If Model B is just Model A with an extra variable, then under the null hypothesis that the extra variable is useless, the two models are identical, and the machinery of the test breaks down. The test becomes badly undersized and you will fail to detect genuine improvements. Clark and West later developed the correction for this case, and if you are comparing a model to a simpler version of itself, you need it.
• It compares the forecasts, not the model-building process. The test evaluates two given forecast series. It says nothing about the fact that you tried forty models before picking these two. If you data-mined your way to Model A, Diebold-Mariano will happily certify it as significantly better than Model B, because it has no idea about the thirty-eight models in the bin. Guarding against that requires an entirely different set of tools.
• You need a reasonable number of forecasts. With very few out-of-sample observations, the test has little power, and its normal approximation is shaky. Small-sample corrections exist and should be used.
• It is a test of unconditional average accuracy. Model A might be better on average but catastrophically worse in exactly the market conditions where being right matters most. The test will not tell you that.

The one-line takeaway

Diebold and Mariano turned the everyday question "is my model actually better?" into a proper statistical test, by racing the two models head to head, differencing away the shared shocks, and asking whether the average margin of victory is distinguishable from zero under assumptions loose enough to survive real financial data.