Ordinary Least Squares (OLS)

Ordinary least squares is the most-used estimator in quantitative finance, factor models, hedge ratios, beta estimation, and most linear signals are OLS underneath. It is worth understanding not as a black box but from its geometry and its optimality theorem, because the places OLS fails are exactly the places financial data lives.

The model

We observe $n$ paired observations of an outcome $y_i$ and a $k$-vector of regressors $x_i$, and assume a linear conditional mean:

$y_i = x_i^\top \beta + \varepsilon_i, \qquad i = 1, \dots, n.$

Stacking rows gives the matrix form $y = X\beta + \varepsilon$, where $y \in \mathbb{R}^n$, $X \in \mathbb{R}^{n \times k}$ (the design matrix, typically with a column of ones for the intercept), $\beta \in \mathbb{R}^k$ is the parameter we want, and $\varepsilon$ is the unobserved error.

OLS chooses $\hat\beta$ to minimize the residual sum of squares:

$\hat\beta = \arg\min_{b} \; \lVert y - Xb \rVert_2^2 = \arg\min_b \sum_{i=1}^n (y_i - x_i^\top b)^2.$

Deriving the estimator

The objective $S(b) = (y - Xb)^\top (y - Xb)$ is convex in $b$. Expanding,

$S(b) = y^\top y - 2 b^\top X^\top y + b^\top X^\top X b.$

Setting the gradient to zero gives the normal equations:

$\frac{\partial S}{\partial b} = -2X^\top y + 2 X^\top X b = 0 \;\Longrightarrow\; X^\top X \, \hat\beta = X^\top y.$

If $X$ has full column rank, $X^\top X$ is invertible and

$\boxed{\hat\beta = (X^\top X)^{-1} X^\top y.}$

The Hessian $2X^\top X$ is positive semidefinite (and positive definite under full rank), confirming a global minimum.

The geometry: projection

The fitted values are $\hat y = X\hat\beta = X(X^\top X)^{-1}X^\top y = Hy$, where $H = X(X^\top X)^{-1}X^\top$ is the hat matrix. $H$ is the orthogonal projection onto the column space of $X$: it is symmetric ($H^\top = H$) and idempotent ($H^2 = H$). So OLS decomposes $y$ into the part explained by $X$ (its projection $\hat y$) and an orthogonal residual $\hat\varepsilon = (I - H)y$. Orthogonality, $X^\top \hat\varepsilon = 0$, is the normal equations, the residual is uncorrelated with every regressor by construction. This is the single most useful mental image of regression: least squares is projection.

The Gauss-Markov assumptions

The estimator above is mechanical; its statistical properties require assumptions.

1. Linearity: $y = X\beta + \varepsilon$ with $\beta$ fixed.
2. Strict exogeneity: $\mathbb{E}[\varepsilon \mid X] = 0$. This is the load-bearing assumption, it delivers unbiasedness and is what "no omitted variables, no reverse causality, no measurement error in $X$" really means.
3. No perfect multicollinearity: $X$ has full column rank $k$.
4. Spherical errors: $\operatorname{Var}(\varepsilon \mid X) = \sigma^2 I_n$, errors are homoskedastic and uncorrelated.

Assumptions 1–3 give unbiasedness; adding 4 gives efficiency.

Properties

Unbiasedness. Substituting $y = X\beta + \varepsilon$,

$\hat\beta = (X^\top X)^{-1}X^\top(X\beta + \varepsilon) = \beta + (X^\top X)^{-1}X^\top \varepsilon.$

Taking expectations conditional on $X$ and using strict exogeneity, $\mathbb{E}[\hat\beta \mid X] = \beta$.

Variance. From the same expression,

$\operatorname{Var}(\hat\beta \mid X) = (X^\top X)^{-1}X^\top (\sigma^2 I) X (X^\top X)^{-1} = \sigma^2 (X^\top X)^{-1}.$

The unknown $\sigma^2$ is estimated unbiasedly by $\hat\sigma^2 = \hat\varepsilon^\top \hat\varepsilon / (n - k)$; the $n-k$ denominator (degrees of freedom) corrects for the $k$ parameters already fit.

Gauss-Markov (optimality). Among all linear, unbiased estimators, OLS has the smallest variance, it is BLUE (Best Linear Unbiased Estimator). Note what this does not say: it says nothing about nonlinear estimators, and under fat tails or heteroskedasticity a biased estimator (ridge) or a robust one can beat OLS in mean-squared error. See The Gauss–Markov Theorem.

Inference

Add normality, $\varepsilon \mid X \sim \mathcal{N}(0, \sigma^2 I)$, and $\hat\beta$ is exactly Gaussian:

$\hat\beta \mid X \sim \mathcal{N}\!\big(\beta,\ \sigma^2 (X^\top X)^{-1}\big).$

The $t$-statistic for a coefficient, $t_j = \hat\beta_j / \operatorname{se}(\hat\beta_j)$, follows a $t_{n-k}$ distribution, giving hypothesis tests and confidence intervals. Even without normality, the CLT delivers asymptotic normality, so these tests hold approximately in large samples.

Goodness of fit

$R^2 = 1 - \dfrac{\hat\varepsilon^\top \hat\varepsilon}{\sum_i (y_i - \bar y)^2}$ is the fraction of variance explained. Two warnings that matter in practice: $R^2$ never decreases when you add regressors (use adjusted $R^2$, which penalizes $k$), and a high $R^2$ is neither necessary nor sufficient for a good model, a tiny $R^2$ is normal and often tradeable in cross-sectional return regressions.

Worked example: estimating beta

Regress a stock's excess returns $r_t$ on the market's excess returns $m_t$: $r_t = \alpha + \beta m_t + \varepsilon_t$. Here the slope has the closed form

$\hat\beta = \frac{\widehat{\operatorname{Cov}}(r, m)}{\widehat{\operatorname{Var}}(m)}, \qquad \hat\alpha = \bar r - \hat\beta \bar m.$

If the sample covariance is $0.018$ and the market variance is $0.020$, then $\hat\beta = 0.9$, the stock moves $0.9\%$ per $1\%$ market move on average. The intercept $\hat\alpha$ is the sample estimate of Jensen's Alpha (α). This single-regressor case shows the general truth: OLS slopes are (co)variance ratios.

Failure modes in financial data

This is where quants earn their keep, every Gauss-Markov assumption is routinely violated by markets:

• Heteroskedasticity. Volatility clusters, so $\operatorname{Var}(\varepsilon_t)$ is not constant. $\hat\beta$ stays unbiased but the standard errors are wrong; use White/robust standard errors. See Heteroskedasticity.
• Autocorrelation. Overlapping returns and momentum induce serially correlated errors, again invalidating naive standard errors (use Newey-West). See Autocorrelation and Serial Correlation.
• Multicollinearity. Correlated factors (value and size, say) make $X^\top X$ near-singular, inflating $\operatorname{Var}(\hat\beta)$ and making coefficients unstable and sign-flippy. See Multicollinearity.
• Endogeneity. If a regressor is correlated with the error, omitted variables, simultaneity, or noisy predictors, strict exogeneity fails and $\hat\beta$ is biased and inconsistent, no sample size saves it. This is the deepest problem and motivates instrumental variables.
• Outliers and fat tails. Squared loss weights large residuals heavily, so a few extreme return days can dominate the fit; robust regression or winsorization helps.
• Overfitting with many regressors. As $k$ grows relative to $n$, in-sample $R^2$ rises mechanically while out-of-sample performance collapses, the motivation for Ridge and LASSO Regularization.

In interviews

Expect to derive $\hat\beta = (X^\top X)^{-1}X^\top y$ from the normal equations, explain why strict exogeneity (not just zero-mean errors) is what you need for unbiasedness, state Gauss-Markov precisely (linear and unbiased, candidates who drop "linear" are wrong), and identify which failures bias the point estimate (endogeneity) versus only the standard errors (heteroskedasticity, autocorrelation). A common follow-up: "when would you not use OLS?", the honest answer names regularization for high-dimensional or collinear designs and robust methods for fat tails.