Gradient Boosting: Building a Great Model Out of a Chain of Bad Ones

Random forests work by building hundreds of trees in parallel, all trying to solve the same problem independently, and averaging away their disagreements.

Gradient boosting works the other way around. It builds trees one after another, in a chain, and each new tree has exactly one job: clean up the mess left by everything that came before it.

That difference in architecture, parallel versus sequential, turns out to matter enormously. Jerome Friedman's 2001 paper laid out the general theory, and the algorithms that descend from it (XGBoost, LightGBM, CatBoost) are, twenty-five years later, still the best thing you can throw at a table of numbers.

The problem: how do you learn a function you cannot write down?

Suppose the true relationship between company characteristics and future returns is some complicated, unknown function. You want to approximate it.

The classical approach: guess the form of the function (a straight line, say), then find the parameters that fit best. This works if your guess is right and fails silently if it is wrong.

Boosting had already been invented as an alternative, and it worked shockingly well, but for years it had the air of a lucky trick. Nobody could fully explain why repeatedly training weak models on reweighted data produced such good results.

Friedman's paper is the one that explained it. And his explanation is what turned boosting from a clever hack into a general engine you could point at any problem.

The key idea via analogy: an apprentice who only studies their own mistakes

Here is the whole algorithm, in plain English.

Start with something stupid. Predict the average return for every stock. This model is useless, but it is a starting point.
Look at what you got wrong. For each stock, compute the difference between what actually happened and what you predicted. These are your residuals, your errors.
Train a small, weak model, a shallow decision tree, to predict those errors. Not the returns. The errors. You are asking: is there any pattern in what I am getting wrong? Am I systematically too low for small-cap stocks? Too high in volatile months?
Add a small fraction of that error-predicting tree to your model. Not all of it. A small fraction. This is called the learning rate, and it is critical, as we will see.
Recompute your errors, which are now slightly smaller. Go to step 2. Repeat a thousand times.

Each tree in the chain is deliberately weak, typically just a few splits deep. On its own it is nearly worthless. But each one is aimed precisely at the residual weakness of the current ensemble. The model climbs, in a thousand small steps, toward an accurate fit.

The analogy: an apprentice who, after every attempt, studies only their mistakes and practises exactly the thing they got wrong, over and over, a thousand times. They never waste effort on what they already do well. That is a ferociously efficient way to learn.

Friedman's actual contribution: the "gradient" part

The word "gradient" is the key to the paper's generality, and it is worth understanding, because it is what makes this a framework rather than an algorithm.

Fitting to the residuals, as described above, is the right move if your goal is to minimize squared error. But what if you do not care about squared error? What if you are doing classification, or you want a model that is robust to outliers, or you want to predict a quantile rather than a mean? What are the "errors" you should be chasing then?

Friedman's insight: the residual is just a special case. What you should really fit each new tree to is the gradient of your chosen loss function, which is the direction in which the loss decreases fastest. For squared error, that gradient happens to work out to be exactly the residual. But for any other loss you care to name, you can compute its gradient, and the algorithm still works.

This is why Friedman called it a machine. He was not describing one algorithm. He was describing a template: pick any loss function you like, compute its gradient, and boost against it. Robust regression, classification, ranking, custom financial objectives that penalise downside losses more than upside misses. All of it, from one framework.

The paper reframes model fitting as gradient descent in function space: instead of taking small steps in parameter space (as you do when training a neural network), you take small steps in the space of functions, and each step is a decision tree.

The two ideas that stop it exploding

Boosting, as described, is a machine for fitting the training data perfectly, including all the noise. Left unchecked it overfits catastrophically. Friedman supplies two safeguards, and both are now universal.

Shrinkage, also called the learning rate. Do not add each new tree at full strength. Add a small fraction of it, maybe five or ten percent. This means you need far more trees to converge, which is slower, but the model creeps toward the answer rather than lunging at it, and it generalizes dramatically better. This is the single most important tuning parameter in boosting, and Friedman's finding that a small learning rate with many trees beats a large learning rate with few trees is one of the most reliably useful results in applied machine learning.

Subsampling, in the related "stochastic gradient boosting" work. Train each tree on a random subset of the data rather than all of it. This injects randomness, decorrelates the trees, and, somewhat counterintuitively, improves accuracy while also speeding things up.

Why it mattered

It is the best general-purpose predictor for tabular data, still. Two and a half decades on, with all the excitement about deep learning, if you hand a professional a spreadsheet of features and ask for the most accurate predictions possible, they will reach for a gradient boosting library. Deep learning conquered images and text. It has not conquered tables.
It gave finance its non-linear workhorse. Predicting returns from firm characteristics is a tabular problem with mediocre signal, correlated features and non-linear interactions. That is boosting's home turf. Gu, Kelly and Xiu found boosted trees among the top performers in their machine learning horse race for stock returns.
It handles custom objectives, which finance genuinely needs. Squared error is rarely what a trader actually cares about. You might want to penalise being wrong in the direction that costs you money more than being wrong in the direction that costs you an opportunity. Friedman's framework lets you write that down as a loss function and boost against it directly.
It spawned the tools everyone uses. XGBoost, LightGBM and CatBoost are engineering achievements built directly on this mathematical foundation. Friedman wrote the theory; Chen and Guestrin and others made it fast.

The honest limitations

It overfits with enthusiasm, and finance is where that hurts most. Boosting is a machine for relentlessly fitting whatever structure is in the training data, and in financial data, most of the structure is noise. Random forests degrade gracefully as noise increases; boosting drives straight into the wall. You need aggressive regularization, honest early stopping, and cross-validation that respects the arrow of time, or your backtest is a work of fiction.
It has a lot of knobs, and every knob is an opportunity to overfit. Learning rate, number of trees, tree depth, subsample fraction, minimum leaf size, and more. Tuning all of them by cross-validation is itself a large search, and a large search on financial data is exactly what Bailey, Borwein, Lopez de Prado and Zhu warned you about.
It is inherently sequential, so it is slow to train. Each tree depends on the one before it, so you cannot parallelise across trees the way you can with a random forest. Modern implementations parallelise within each tree, which helps, but the fundamental chain remains.
It is a black box, and a deep one. A thousand chained trees, each correcting the last, is not something you can explain to an investment committee. Feature importance scores help a little and mislead in the presence of correlated features, which in finance is always.
It cannot extrapolate. Like any tree-based method, its predictions are averages of training labels, so it can never predict a value outside the range it has seen. A genuinely unprecedented market is a market it has no way to reason about.

The one-line takeaway

Friedman showed that you can build an extremely accurate model by chaining together hundreds of deliberately weak models, each one trained solely to correct the errors of everything before it, and by framing that process as gradient descent in the space of functions, he made it work for any objective you can write down.