Random Forests: Why a Crowd of Deliberately Weakened Trees Beats One Smart One

A decision tree is the most intuitive predictor ever invented. It is a flowchart. Is the stock's price-to-book ratio below 1.2? If yes, is its momentum positive? If yes, is its volatility low? Then predict a high return. You could explain it to anyone.

Decision trees have exactly one problem, and it is fatal: they are wildly unstable. Change a handful of data points and the tree that comes out the other side can be completely different, with different splits in a different order. A single tree grown deep enough to be useful will also happily memorize every quirk of your training data and then fall apart on anything new.

Leo Breiman's solution, published in 2001, is one of the strangest and most effective ideas in machine learning. He fixed the instability by making it worse, on purpose, and then averaging.

The problem: high variance, low bias

Statisticians describe the failure mode of a deep decision tree as high variance. A high-variance predictor is one whose answer swings around dramatically depending on which sample of data it happened to see. It is not systematically wrong in any particular direction, it is just erratic. It fits the training data superbly, including all the noise, and therefore does badly on new data.

The classical remedy is to constrain the model: prune the tree, limit its depth, force it to be simpler. This trades variance for bias: the model is now steadier but too crude to capture the real structure. You are picking your poison, and neither poison is nice.

Breiman found a third option.

The key idea via analogy: a committee of eccentric experts

Suppose you want to estimate the number of jellybeans in a jar. You could ask one very careful expert. Or you could ask a thousand random people and take the average of their guesses.

The famous "wisdom of crowds" result says the crowd usually wins. But it wins only under a specific condition: the individual errors have to be roughly independent. If everyone in the crowd guesses too high, averaging does not help, because averaging a thousand identical errors just gives you the error back. Averaging only cancels out mistakes when the mistakes point in different directions.

Now apply that to decision trees. If you grow a thousand trees on the same data, you get a thousand identical trees, and averaging them gives you nothing. Their errors are perfectly correlated.

So Breiman's problem became: how do I force my trees to disagree?

His answer has two mechanisms, and the second one is the real invention.

Mechanism one: bagging. Train each tree on a different bootstrap resample of the data, that is, a random draw with replacement from your original dataset. Each tree now sees a slightly different world. This idea was Breiman's too, from a few years earlier, and it helps. But it does not help enough. If one predictor is genuinely very strong (say, momentum), then every tree, no matter what resample it sees, will pick momentum as its first split. The trees remain far too similar.

Mechanism two, the key one: at every single split, hide most of the features from the tree. When a tree is deciding what question to ask at a given node, do not let it look at all the predictors. Show it only a small random subset, and force it to make the best split it can using only those. At the next node, draw a fresh random subset.

This is a bizarre thing to do. You are deliberately crippling each tree, withholding information, forcing it to make worse decisions than it could. Each individual tree in a random forest is worse than a tree you would have built by hand.

And yet the forest is better. Why? Because the crippling is exactly what breaks the correlation. Now, when momentum is hidden from a particular node, the tree is forced to find some other useful structure, and it discovers relationships that a greedy tree would have steamrolled past. The trees become genuinely diverse. Their errors point in different directions. And when you average them, the errors cancel while the signal, which they all agree on, survives.

Breiman traded a small amount of individual accuracy for a large amount of decorrelation, and the trade is overwhelmingly profitable. That is the whole paper.

The free lunch: out-of-bag error

There is a lovely bonus that falls out of the design. Because each tree is trained on a bootstrap resample, roughly a third of the data is left out of any given tree's training set. Those left-out observations are, for that tree, a genuine held-out test set.

So you can estimate the forest's generalization error without a separate validation set at all: for each data point, ask only the trees that never saw it. This is called the out-of-bag error, and it means a random forest tells you how good it is, for free, as a side effect of how it was built.

Why it mattered

• It works, immediately, on almost anything. Random forests are famously the strongest thing you can get with the least effort. Little tuning, no feature scaling, robust to outliers, handles mixed data types, does not fall over with more features than observations. For twenty years the professional advice for any new prediction problem has been "run a random forest first and see what you get."
• It gave finance a non-linear tool that does not immediately overfit. This is a big deal in a field where the signal-to-noise ratio is brutal. Forests capture interactions between features automatically, which linear regressions cannot, without the runaway overfitting that a single deep tree would produce. Gu, Kelly and Xiu found trees among the best performers in their machine learning horse race for stock returns.
• It made variable importance routine. The forest can measure how much each feature contributes, by checking how much accuracy degrades when you scramble that feature. It is a crude tool, and it can mislead, but it is far better than the nothing you get from a neural network.
• It cemented ensembling as a first-class idea. The lesson that many weak, diverse models beat one strong one now underpins gradient boosting, model stacking, and much of modern applied machine learning.

The honest limitations

• It is still a black box, just a more polite one. A single decision tree is interpretable. Five hundred averaged trees are not. Variable importance scores give you a rough sense of what matters, but they are unreliable when features are correlated (which, in finance, they always are), and they tell you nothing about the shape of the relationship.
• It cannot extrapolate. At all. A forest's prediction is always an average of training labels, so it can never predict a value outside the range it has seen. For a market entering a genuinely unprecedented regime, this is a real and dangerous limitation.
• The default version has no idea time exists. Bagging assumes your observations are exchangeable, which means it will cheerfully build trees that train on 2020 data and predict 2015. Applied naively to financial time series, a random forest will leak the future into the past and produce a backtest that is pure fantasy. You have to impose the time structure yourself, with proper walk-forward or purged cross-validation.
• Noise is its enemy, and finance is mostly noise. Forests do very well when there is real structure to find. When the signal-to-noise ratio is as bad as it is in return prediction, even a forest will spend most of its capacity fitting noise. It degrades gracefully, but it degrades.
• Boosting usually beats it now. For pure predictive accuracy on tabular data, gradient boosted trees (XGBoost and its descendants) have generally overtaken random forests. The forest's advantage is that it is much harder to misuse.

The one-line takeaway

Breiman showed that the way to fix an unstable predictor is not to make it more careful but to build hundreds of them, deliberately hobble each one by hiding most of the features at every split so they are forced to disagree, and average the results, because averaging cancels errors only when the errors are different.