Nudge One Number, Rebuild the Whole Portfolio: Best and Grauer on Optimizer Fragility

Michaud told us mean-variance optimizers are "estimation-error maximizers." That is a memorable phrase and an accurate diagnosis, but it is not a measurement. How sensitive is an optimized portfolio, exactly? If I change one asset's expected return by a tenth of a percent, does the portfolio wobble slightly, or does it detonate?

Michael Best and Robert Grauer answered that question properly in 1991, with analysis rather than anecdote. They derived the mathematics of how efficient portfolio weights respond to changes in the inputs, and then computed what happens with real data.

The answer is: it detonates.

The problem: a machine whose output is unstable is not a machine you can use

A tool that gives wildly different answers in response to imperceptibly different inputs is useless, no matter how correct it is. Engineers call this ill-conditioning, and they treat it as a defect even when the underlying math is exact.

Portfolio managers had been complaining for decades that optimizers spat out absurd, unstable portfolios. But complaints are easy to dismiss as user error. What Best and Grauer supplied was a rigorous account of exactly why the instability arises and exactly how large it is.

The key idea via analogy: the see-saw balanced on a needle

Picture a see-saw balanced on a very sharp fulcrum. It is in equilibrium. Now imagine the plank is nearly frictionless and extremely long. The lightest touch on one end sends it swinging violently.

That is a mean-variance frontier. The optimizer is finding the balance point of a system where the assets are highly correlated with each other, meaning many different portfolios have nearly identical risk-return characteristics. When many combinations are nearly equally good, the surface the optimizer is walking on is nearly flat. And when the surface is nearly flat, a tiny tilt in the inputs sends the answer sliding a very long way.

Concretely, suppose two assets are 95 percent correlated. The optimizer sees them as near-substitutes. If your estimate says asset A will return 8.0 percent and asset B will return 7.9 percent, the optimizer will go long A heavily and short B heavily to harvest that tiny apparent difference, since it can construct a nearly risk-free spread between two near-identical assets and the spread has a small positive expected return. Now change your estimate so B returns 8.1 percent instead. The optimizer reverses the entire position. A one-fifth of one percent change in an input flips the portfolio inside out.

Best and Grauer showed this analytically. When the only constraint on the optimizer is that the weights sum to one, meaning shorting is unrestricted, a small change in a single asset's expected return can drive large swings in the weights, and in the resulting portfolio's mean and variance. Positions do not just adjust. Assets can go from large long positions to large short positions, and whole sets of securities can drop out of the portfolio entirely.

Consider what this means for how you should feel about an optimizer's output. The portfolio it hands you is presented with the authority of mathematics: this is the optimal portfolio. But if a change to your inputs that is far smaller than your estimation error would produce a completely different portfolio, then the specific portfolio you were given carries essentially no information. You cannot distinguish it from the many other portfolios you would have gotten with equally plausible inputs.

The other half: constraints tame it

Best and Grauer did not only deliver bad news. They also showed something practically vital: imposing constraints dramatically reduces the sensitivity.

Forbid short selling and the optimizer can no longer construct those enormous offsetting long-short spreads between near-substitute assets. The weights are penned in. A small change in an input can now only move the portfolio so far, because the constraints form walls. Add upper bounds on individual weights and it gets tamer still.

There is something philosophically odd going on here that later authors would develop. You are restricting the optimizer, which by definition means the constrained portfolio is worse than the unconstrained one in terms of the objective as measured with your assumed inputs. And yet the constrained portfolio is better in the real world, because your inputs are wrong and the constraints stop the optimizer from acting too confidently on them. Jagannathan and Ma would later show precisely why: imposing a no-short-sale constraint is mathematically equivalent to shrinking your covariance estimates. A constraint is a disguised form of humility about your inputs.

Why it mattered

• It made the fragility a fact rather than a folk complaint. After Best and Grauer, "optimizers are unstable" was not a practitioner grumble, it was a derived property with known drivers.
• It located the culprit: the expected returns. The sensitivity is dominated by the means, which is precisely the input we estimate worst. Chopra and Ziemba's finding, that errors in means are roughly an order of magnitude more costly than errors in variances, is the natural companion result.
• It provided the theoretical basis for constrained optimization as standard practice. Essentially every real-world portfolio optimizer runs with long-only constraints, position limits, sector limits and turnover limits. This paper is a big part of why that is understood to be a feature rather than a compromise.
• It supports the entire shrinkage and Bayesian agenda. If the optimizer overreacts to differences in the means, then compressing the means together (Jorion, Black-Litterman) directly addresses the disease at its source.

The honest limitations

• It is a critique, not a construction. The paper tells you the machine is fragile and that constraints help. It does not tell you what the right constraints are, and there is no theory that does. In practice constraints are chosen by judgment and convention.
• Constraints have a cost you cannot see. By penning in the optimizer, you also prevent it from acting on information that is genuinely correct. If you truly do have a strong view, a long-only constraint means you cannot express it. You are trading away real alpha in exchange for protection against imagined alpha, and you cannot tell which is which in advance.
• The results are about a specific setup. The analysis concerns mean-variance efficient portfolios under a budget constraint, with the extensions to inequality constraints. Real optimizers have more machinery, and the sensitivity in practice depends on the particular asset set and correlation structure.
• The extreme sensitivity is worst when assets are near-substitutes. For portfolios of genuinely distinct asset classes with modest correlations, optimizers are better behaved. The problem is at its most vicious in exactly the case people care about most: large universes of correlated equities.

The one-line takeaway

Best and Grauer proved that a mean-variance portfolio's weights can swing violently in response to a change in expected returns far smaller than your estimation error, which means the specific portfolio an unconstrained optimizer hands you is not really a recommendation, it is one arbitrary draw from a huge cloud of equally plausible answers, and that adding constraints, though it looks like a compromise, is what makes the whole exercise usable.