Why piling on predictors quietly inflates your variance

Question

A junior researcher keeps adding predictors to a linear model because the training $R^2$ keeps climbing.

Explain, in bias–variance terms, why adding regressors mechanically improves in-sample fit but can wreck out-of-sample performance.

Accepted Answer

In-sample fit can only improve. Least squares chooses coefficients to minimize the residual sum of squares. Adding a column gives the optimizer strictly more freedom, so the minimized RSS can never rise, even if the new predictor is pure noise, it will be assigned a small nonzero coefficient that soaks up a little of the residual. Hence training error falls and R^2 rises with every feature. That is not evidence the feature helps. The hidden cost is variance. With p regressors and noise variance \sigma^2, the model spends p degrees of freedom fitting the sample. A useful benchmark: the expecte…

Why piling on predictors quietly inflates your variance

Your answer

More Statistics questions