Better to Fail Conventionally: Herd Behavior and Career Risk

Keynes wrote that "worldly wisdom teaches that it is better for reputation to fail conventionally than to succeed unconventionally." It is one of the great lines in finance, and for fifty years it sat there as a cynical aphorism that nobody had modelled.

In 1990, Scharfstein and Stein modelled it. And the result is not a story about weak-willed managers who lack the courage of their convictions. It is a story about managers who behave exactly rationally given their incentives, and whose incentives happen to punish independent thought.

This paper is the reason your fund manager owns what everyone else owns.

The problem: you are not being judged on your returns

Start with the observation that breaks the standard model.

A fund manager, in theory, is judged on performance. Make money, keep your job. So the manager should simply do whatever maximizes expected return, using the best information available.

But that is not how it works, and everyone in the industry knows it. The manager is judged on performance, yes, but also on something else: whether they seem competent. And "seeming competent" is evaluated by people (clients, bosses, trustees, boards) who cannot directly observe whether a decision was smart, and can only observe the outcome and the context.

Scharfstein and Stein's key insight is about how an observer infers ability from outcomes, and it turns entirely on one thing: whether other people made the same call.

The key idea via analogy: the four ways to be judged

Imagine you are a fund manager. There are exactly four possible outcomes, and their consequences are wildly asymmetric.

1. You follow the crowd, and the crowd is right. Great. Solid performance, everyone is happy, you look competent. A good outcome.

2. You follow the crowd, and the crowd is wrong. You lost money. But so did everybody else. And what does your boss conclude? Not "you are an idiot", but "this was an unforeseeable event that fooled the entire industry, nobody could have known." Your reputation is almost completely protected. You are shielded by the herd. This is the crucial cell of the matrix, and it is the paper's engine: shared blame is nearly costless.

3. You break from the crowd, and you are right. Excellent. You look like a genius. Big upside.

4. You break from the crowd, and you are wrong. Catastrophe. You made a bet nobody else made, and it failed. There is no excuse available. Nobody else can be blamed, no environment can be blamed. The only explanation is that you personally are incompetent, reckless, or both. You are fired.

Now compare the payoffs. Breaking from the herd has a big upside and a catastrophic downside. Following the herd has a modest upside and almost no downside, because being wrong with everyone else is treated as bad luck rather than bad judgment.

Given that asymmetry, what does a rational, career-maximizing manager do? They herd. Even when their own research screams that the crowd is wrong.

Scharfstein and Stein formalize this with a mechanism that gives the model real bite. The observer (the boss, the client) is trying to figure out whether the manager is smart or dumb. The trick is in how these two types differ:

• Smart managers get signals correlated with the truth, and therefore correlated with each other. If two smart managers look at the same company, they will tend to reach the same conclusion, because they are both seeing something real.
• Dumb managers get noise, which is uncorrelated with anything, including each other.

Now think about what the boss infers when he sees two managers agree. Agreement is evidence of smartness, because only the smart types systematically agree. And what does he infer when he sees a manager go it alone and fail? Disagreement plus failure is evidence of dumbness, because that is exactly what noise looks like.

So the manager who copies is manufacturing a signal of competence, regardless of whether the copied decision is any good. Herding is not a failure of nerve. It is a rational investment in reputation, and the model shows it can be optimal even when the manager privately knows the herd is heading off a cliff.

The consequence, stated plainly by the paper: managers discard their own private information and mimic the decisions of others. Which means the information they worked so hard to gather never reaches the price. It dies inside the fund, unexpressed, because expressing it was too dangerous.

Why it mattered

• It explains herding among professionals, which the other theories struggle with. Informational cascade models (Bikhchandani, Hirshleifer and Welch) explain herding among people who genuinely learn from each other's actions. Scharfstein and Stein explain herding among people who know better and copy anyway. That is a darker and, for finance, a more relevant mechanism, because professional investors emphatically do know better.
• It is a limits-to-arbitrage paper in disguise. Shleifer and Vishny would later show that arbitrageurs cannot correct mispricing because their capital flees. Scharfstein and Stein show they may not even try, because their careers are at stake. Same conclusion, different channel: the smart money is structurally constrained from doing the smart thing. Both papers say the mispricing survives not because nobody knows, but because knowing is not enough.
• It explains why everyone owns the same stocks. Benchmark-hugging, closet indexing, and the crowding of professional portfolios into the same names are all direct predictions of this model. A manager who deviates from the benchmark is taking career risk that is not compensated by the fee structure. So they don't.
• It explains why bubbles get professional participation. The standard defence of markets is that professionals would never buy into a mania. This paper says they would, and rationally so. In 1999, a fund manager who refused to own tech stocks and watched his benchmark run away from him got fired long before he was vindicated. Several famously did. The model predicts precisely that.

The honest limitations

• The model is stylized and the mechanism is specific. It requires a particular structure: the observer's inference problem, the correlated-signals assumption, a two-period setup. Real reputation formation is messier and involves many more signals than a single investment decision.
• The correlated-signals assumption is doing a lot of work. The whole result rests on smart managers agreeing with each other and dumb ones not. That is plausible but it is an assumption, and it is not obviously true if smart managers have different good information (in which case they should disagree, and disagreement would be a sign of research effort rather than incompetence).
• Contracts could in principle fix this. If herding is caused by bad incentives, then better incentives should reduce it. Pay managers on long-horizon absolute performance rather than short-horizon relative performance, and the calculus changes. Some funds do exactly this. That such contracts are rare is itself interesting, and the paper does not fully explain why the market has not solved its own problem.
• Distinguishing herding from correlated information is hard. If two managers buy the same stock, is that herding, or did they both do good research and reach the same correct conclusion? From the outside these are observationally identical, which makes empirical testing of this model genuinely difficult, and much of the follow-up work has had to be clever rather than direct.

The one-line takeaway

Scharfstein and Stein showed that professional managers rationally throw away their own research and copy each other, because being wrong alongside everybody else is forgiven as bad luck while being wrong alone is punished as incompetence, which means the market never hears what the independent thinkers actually knew.