The Elastic Net: What to Do When Your Predictors Are All Saying the Same Thing

The lasso was a triumph. Add a penalty on the sum of the absolute values of your regression coefficients, and the model automatically zeroes out the variables that do not earn their keep. Overfitting control and variable selection, from a single line of maths.

Then people started using it on real data, and a nasty behaviour emerged.

When two predictors are highly correlated, the lasso picks one of them and throws the other away, and which one it picks is close to arbitrary.

In genomics, where Hui Zou and Trevor Hastie were working, this was a disaster: genes operate in correlated pathways, and a method that reports "gene 4,171 matters and gene 4,172 does not" when the two are nearly identical is not just unhelpful, it is actively misleading. In quantitative finance, the problem is, if anything, worse, because correlated predictors are not an occasional nuisance. They are the entire dataset.

The problem: three flaws in the lasso

Zou and Hastie identify three specific defects, and the third is the one that hurts.

One: a hard ceiling on how many variables it can select. If you have more predictors than observations, the lasso can never select more variables than you have observations. This is a structural limitation of the geometry, not a tuning problem. In a setting with a thousand candidate features and three hundred months of data, this bites.

Two: it behaves erratically with grouped predictors. More on this below.

Three, and worst: when predictors are highly correlated, the lasso's variable selection is unstable, and it is often outperformed by plain old ridge regression. That is a humbling result. The whole selling point of the lasso is that it does variable selection, which ridge cannot. But in exactly the regime that matters most in practice, the correlated one, the lasso's selections are noise-driven and its predictions are worse than the older, cruder method.

The key idea via analogy: keep the whole family, or drop the whole family

Think about what "value" means as a stock characteristic. There are many ways to measure it: book-to-price, earnings-to-price, cash-flow-to-price, sales-to-price, and several more. They are all telling you roughly the same thing, and they are correlated at maybe 0.8 or 0.9 with each other.

Now suppose value genuinely predicts returns. What should a good variable-selection method do?

The sensible answer, obviously, is: keep them all, and split the credit among them. They are all measuring the same real thing. Each one is a slightly noisy view of it. Averaging across them gives you a cleaner signal than any one alone.

What the lasso actually does: it picks whichever of the six happened to have the marginally best fit in your sample (which is basically a coin flip driven by noise), gives it a large coefficient, and sets the other five to exactly zero. Add two more years of data and it might pick a different one. Your "discovery" that book-to-price is the key value measure and cash-flow-to-price is worthless is a complete fabrication of the sampling noise.

Ridge regression, meanwhile, does something much more sensible with this group. Because its penalty is on squared coefficients, and squaring punishes big numbers disproportionately, ridge strongly prefers to spread weight evenly across correlated predictors rather than concentrate it. It gives all six value measures a similar, moderate coefficient. That is exactly the behaviour you want.

But ridge cannot set anything to zero, so it also keeps the four hundred genuinely worthless predictors, with small non-zero weights.

So the lasso has the selection you want and the wrong grouping behaviour. Ridge has the grouping behaviour you want and no selection.

Zou and Hastie's answer, and once stated it seems inevitable: use both penalties at once. Penalize the sum of absolute values (which produces zeros) and the sum of squares (which encourages grouping). Blend them, with a knob controlling the mix.

They call it the elastic net, and the metaphor is apt: it is a stretchy net that catches whole schools of correlated fish together rather than plucking out one and letting the rest swim away.

The property they prove is called the grouping effect: under the elastic net, predictors that are highly correlated with each other get assigned similar coefficients. They go into the model together or they go out together. That is the behaviour you actually wanted from the start.

The bit that could have sunk it

There is a subtlety the paper handles carefully. Applying two shrinkage penalties at once means you are shrinking your coefficients twice, and the result is a model that is over-shrunk: everything is too small, and predictions suffer.

Zou and Hastie identify this (they call it the double shrinkage problem) and correct for it with a rescaling step. Without that correction, the naive combination of the two penalties performs disappointingly, and the method might have been dismissed. It is a good example of a paper whose value lies not just in the idea but in the care taken to make the idea actually work.

Why it mattered

• It is the right default for financial data. In finance, predictors are correlated by construction. There are twenty momentum measures, a dozen value measures, half a dozen quality measures. Any method that arbitrarily picks one from each family and calls the rest useless is producing an illusion of insight. The elastic net does the sane thing.
• It made high-dimensional selection stable. Run the lasso on two different samples and you may get two different variable lists. Run the elastic net and you tend to get consistent groups. Stability is not a luxury; a variable list that changes every quarter is not telling you anything about the world.
• It broke the variable-count ceiling. The elastic net can select more variables than you have observations, which the lasso cannot. In genuinely high-dimensional problems this is a hard requirement, not a nicety.
• It became standard equipment. The elastic net is one of the main contestants in Gu, Kelly and Xiu's machine learning horse race for stock returns, and it is the default penalized regression in most practitioners' toolkits. It is in every major statistics library, usually as the general case with lasso and ridge as special settings of a single parameter.

The honest limitations

• You now have two knobs, not one. You must choose the overall strength of the penalty and the mix between the two penalty types. That is a two-dimensional cross-validation search, which costs more computation and, more importantly, gives you one more thing to accidentally overfit. In financial time series, where honest cross-validation is already difficult, this is not a trivial cost.
• The grouping effect is a hypothesis about the world, not a fact. The elastic net assumes that correlated predictors should share credit. Usually they should. But sometimes two correlated variables really do have genuinely different relationships with the outcome, and forcing them to have similar coefficients is imposing a falsehood. The method's central virtue is also a built-in assumption.
• It is still linear. Everything the elastic net does happens inside a straight-line model. Non-linear relationships and interactions between features are invisible to it, no matter how beautifully you tune the penalties. This is the same wall the lasso and ridge hit.
• Post-selection inference is still a minefield. As with the lasso, you cannot select variables with the elastic net and then report ordinary p-values on the survivors. The selection step is a search, and searching contaminates inference.
• Interpretability improves but does not arrive. A model that keeps six correlated value measures with moderate coefficients each is more honest than one that keeps one with a big coefficient, but it is not obviously easier to explain to a risk committee.

The one-line takeaway

Zou and Hastie fixed the lasso's worst failure, that it arbitrarily picks one variable from a group of correlated ones and zeroes out the rest, by combining the lasso's penalty with ridge's penalty, producing a method that selects variables in coherent groups rather than plucking single survivors out of noise.