Don't Tell the Test Where to Look: Zivot and Andrews

In 1989 Pierre Perron made a striking claim: the widely believed finding that macroeconomic series are random walks was an artifact of ignoring a couple of enormous historical events. Allow for a level shift at the 1929 crash and a slope change after the 1973 oil shock, and most of those series turned out to be perfectly well-behaved and mean-reverting after all.

It was a compelling argument. Three years later, Eric Zivot and Donald Andrews published a paper whose title starts with "Further Evidence," which in academic English usually means "we have a problem with your paper."

Their objection is one of the cleanest illustrations of data snooping in the entire econometrics literature, and it is a lesson that every quant researcher needs burned into their brain.

The problem: where did the break date come from?

Perron's method requires you to specify when the break happened. He specified 1929 and 1973. Why those dates?

Because he looked at the data and those were the obvious kinks.

Zivot and Andrews put their finger on exactly what is wrong with that. When Perron computed his significance levels, the mathematics assumed the break date was known in advance, handed down from outside the data, as an exogenous fact. But it was not. It was chosen by looking at the series and picking the most dramatic-looking moment.

This changes everything, and it changes it in a direction that flatters your conclusion.

The key idea via analogy: the archer who paints the target afterwards

There is a classic name for this fallacy: the Texas sharpshooter. A man fires his rifle at the side of a barn, walks up, finds the tightest cluster of holes, and paints a bullseye around it. He then announces that he is an excellent shot.

The holes are real. The bullseye is not. It was placed after seeing where the bullets landed, so it carries no evidential weight at all.

Now apply this to Perron's test. If you scan the entire history of a series, find the single most break-like moment in it, and then run a test asking "is there a break here?", of course the test says yes. You chose that point because it looked like a break. The test's significance threshold was designed for someone who nominated the date blindly. It is completely wrong for someone who nominated the date after searching for the best candidate.

The consequence is precisely the one that should worry you: the test rejects the unit root far too often. Perron's conclusion, that most macro series are stationary once you allow for a break, was at least partly manufactured by the way he found the break.

The fix: let the test do the searching, and charge it for the privilege

Zivot and Andrews's solution is the right one, and it has a beautiful logic.

Do not hand the test a date. Make the test find the date itself, and then make it pay for the search.

Concretely: run Perron's break-adjusted unit root test at every possible break date in the sample. Every single one. For each candidate date, you get a test statistic. Then take the date where the evidence against the unit root is strongest, which is exactly what a sharpshooter would do.

Here is the crucial step. Because you deliberately searched for the most favourable break date, you now know you are biased toward rejecting. So Zivot and Andrews derived new, much stricter critical values that account for the fact that you looked everywhere before choosing. The threshold for declaring "this is not a random walk" gets substantially higher.

The elegance is that they made the sharpshooting explicit and priced. They did not forbid the search. They quantified how much the search inflates your chance of a false positive, and raised the bar accordingly. You can still find a break. You just have to clear a much taller hurdle to claim it.

What they found

With the break date estimated rather than assumed, and with the honest critical values applied, the evidence against unit roots got noticeably weaker. For a number of the series where Perron had confidently rejected the random walk, Zivot and Andrews could not. The picture became far more equivocal.

The truth, then, sits somewhere uncomfortable. Perron was right that ignoring breaks corrupts the test. Zivot and Andrews were right that hunting for breaks corrupts it too, in the opposite direction. Neither position gives you a clean answer, and the honest conclusion is that with the data available, we do not confidently know whether many macro series wander or revert. That is unsatisfying and it is also probably correct.

Why it mattered

• It is the definitive demonstration of data snooping in time-series testing. The critical values change because you searched. That is the whole lesson, stated with total clarity in a setting where you can compute exactly how much it matters.
• It became the standard break test. The Zivot-Andrews test is a built-in function in econometrics packages and is the default when you suspect a break but do not want to nominate its date.
• The lesson generalises to every backtest ever run. This is why the paper belongs on a quant's reading list rather than only an economist's. When you scan a hundred parameter settings and report the best Sharpe ratio, you are Perron nominating 1929. When you scan for the best entry threshold, the best lookback window, the best universe filter, and then report a t-statistic computed as though you had chosen those blindly, you are painting a bullseye around your bullet holes. The correct response is not to stop searching. It is to raise the bar in proportion to how hard you searched. That principle, spelled out here in 1992, is the same principle behind the deflated Sharpe ratio and every other modern multiple-testing correction in finance.

The honest limitations

• It still allows only one break. If the series has two or three genuine regime changes, the single-break framework is still misspecified. Bai and Perron later tackled the multiple-break case.
• The alternative hypothesis is asymmetric in an awkward way. In the standard Zivot-Andrews setup, the break is permitted under the "stationary" alternative but not under the "random walk" null. If the truth is a random walk with a break, the test can behave strangely and reject when it should not. Later work refined this.
• Estimated break dates are noisy. The test gives you a most-likely break date, but the uncertainty around that date can be enormous, especially when the break is small. Treating the estimated date as if it were known is a fresh version of the same original sin.
• Power is low, as always. Making the critical values stricter is the right thing to do, and it necessarily makes the test less able to detect real effects. There is no free lunch here: honesty about your search costs you sensitivity. That is a trade you should make, but it is a trade.
• It does not help in real time. The test scans the whole sample to find the break. A trader in the middle of a possible regime change has no future data to scan.

The one-line takeaway

Zivot and Andrews showed that choosing where to look for a break after examining the data is a form of cheating that quietly guarantees you will find one, and their fix, let the test search everywhere but raise the bar in proportion to the search, is the same discipline every honest backtest requires and almost no backtest applies.