Singular Value Decomposition

The singular value decomposition is the most important matrix factorization there is, because it is the one that always exists, for any matrix, of any shape, singular or not. Where Eigenvalues & Eigenvectors work only for square (and behave cleanly only for symmetric) matrices, the SVD handles the rectangular, rank-deficient, noisy data matrices that finance actually produces. It is simultaneously the engine of PCA, the optimal tool for compression and denoising, and the numerically stable way to solve least squares. If you learn one decomposition deeply, learn this one.

The theorem

Every matrix $A \in \mathbb{R}^{m\times n}$ factors as $A = U\Sigma V^\top,$ where $U \in \mathbb{R}^{m\times m}$ and $V \in \mathbb{R}^{n\times n}$ are orthogonal ($U^\top U = I$, $V^\top V = I$) and $\Sigma \in \mathbb{R}^{m\times n}$ is diagonal with non-negative entries $\sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_r > 0$ (and zeros after), the singular values. The columns of $U$ are the left singular vectors, the columns of $V$ the right singular vectors, and $r = \operatorname{rank}(A)$ is the number of nonzero singular values. Existence is unconditional, no assumptions on $A$ at all.

Relation to eigen-decomposition. The SVD is the eigen-decomposition of the symmetric Positive Semidefinite Matrices matrices $A^\top A$ and $AA^\top$ in disguise: $A^\top A = V\Sigma^\top\Sigma\, V^\top, \qquad AA^\top = U\Sigma\Sigma^\top U^\top.$ So the right singular vectors $V$ are eigenvectors of $A^\top A$, the left singular vectors $U$ are eigenvectors of $AA^\top$, and the singular values are the square roots of their shared nonzero eigenvalues, $\sigma_i = \sqrt{\lambda_i}$. This is why singular values are always real and non-negative, they come from a PSD matrix. It also warns you numerically: forming $A^\top A$ squares the condition number, so a good SVD algorithm never actually builds it (it works on $A$ directly via Golub–Kahan bidiagonalization).

The geometry: rotate, stretch, rotate

The SVD says every linear map, however complicated, is just three simple operations in sequence. Reading $Ax = U\Sigma V^\top x$ right to left: $V^\top$ rotates the input into the right singular basis, $\Sigma$ stretches each axis by its singular value (and projects out the null directions), and $U$ rotates into the output basis. A matrix maps the unit sphere to an ellipsoid whose semi-axes have lengths $\sigma_i$ pointing along the columns of $U$. The largest singular value $\sigma_1$ is the maximum stretch (the spectral norm $\lVert A\rVert_2$), the smallest nonzero one is the minimum stretch, and their ratio $\sigma_1/\sigma_r$ is the condition number, the amplification of relative error, the single number that predicts whether solving $Ax=b$ will be stable or a disaster.

Low-rank approximation: Eckart–Young

Here is the result that makes SVD indispensable for data. Truncate the decomposition to the top $k$ terms: $A_k = \sum_{i=1}^k \sigma_i u_i v_i^\top.$ The Eckart–Young–Mirsky theorem says $A_k$ is the best possible rank-$k$ approximation of $A$ in both the spectral and Frobenius norms: $\min_{\operatorname{rank}(B)\le k}\lVert A - B\rVert_F = \lVert A - A_k\rVert_F = \sqrt{\sum_{i=k+1}^{n}\sigma_i^2}.$ No rank-$k$ matrix approximates $A$ better than truncating its SVD, and the error is exactly the tail of discarded singular values. This is the mathematical justification for PCA (the top $k$ singular directions of a demeaned return matrix are the principal components capturing the most variance), for denoising (small singular values are typically noise; dropping them cleans the signal, the basis of random-matrix-theory covariance cleaning), and for compression (store $k(m+n)$ numbers instead of $mn$). The decay profile of the $\sigma_i$ tells you the intrinsic dimensionality: a few large singular values followed by a long flat tail means the data is effectively low-rank plus noise, the empirical signature of factor structure in returns.

The pseudo-inverse and least squares

When $A$ is not invertible (rectangular, or rank-deficient, exactly the Ordinary Least Squares (OLS) problem with collinear regressors), the SVD delivers the Moore–Penrose pseudo-inverse: $A^+ = V\Sigma^+ U^\top, \qquad \Sigma^+ = \operatorname{diag}(1/\sigma_1,\dots,1/\sigma_r,\,0,\dots,0),$ inverting only the nonzero singular values and leaving the null directions at zero. Then $\hat x = A^+ b$ is the least-squares solution, and when the solution is non-unique it selects the minimum-norm one. This reproduces $\hat\beta = (X^\top X)^{-1}X^\top y$ when $X$ has full rank but, crucially, keeps working when $X^\top X$ is singular, where the normal equations simply have no inverse. Truncated SVD (zeroing tiny singular values before inverting) is regularization: it prevents the $1/\sigma_i$ blow-up from near-zero singular values that would otherwise amplify noise catastrophically, the same medicine as ridge regression, applied through the spectrum.

Worked example, compressing a correlation structure

Suppose a return matrix $R$ (days × assets) has singular values $\sigma = (12, 8, 1.1, 0.9, 0.8, \dots)$, a large gap after the second. The energy is $\sum\sigma_i^2$; the top two capture $\frac{12^2 + 8^2}{\sum\sigma_i^2} = \frac{208}{208 + (1.1^2 + 0.9^2 + \dots)}$, the overwhelming majority. Truncating to $A_2 = \sigma_1 u_1 v_1^\top + \sigma_2 u_2 v_2^\top$ retains almost all the variance while collapsing the asset universe to two factors, say a market factor and a sector spread, and treats the remaining near-equal small singular values as noise. This is exactly how a statistical risk model is built and how PCA hedging picks the factors worth neutralising; the sharp drop from $8$ to $1.1$ is the data telling you there are two real factors, not five.

Failure modes and subtleties

• Sign and rotation ambiguity. Singular vectors are defined up to sign, and if singular values tie, up to rotation within the shared subspace. Loadings can flip between samples; that is convention, not instability.
• Centering and scaling matter for PCA. The SVD of a raw return matrix is not PCA, you must demean (and usually standardize) first, or the first singular vector just captures the mean/level. Mixing units without scaling lets the largest-variance asset dominate spuriously.
• Truncation rank is a modelling choice. Where you cut the spectrum trades signal for noise; too few factors underfit risk, too many refit noise. Use the eigenvalue/singular-value gap, cross-validation, or the Marchenko–Pastur noise threshold rather than a fixed number.
• Never form $A^\top A$ to get the SVD. It squares the condition number and loses half your significant digits; use a direct SVD routine.
• SVD is expensive at scale. Full SVD is $O(mn\min(m,n))$; for the top-$k$ of a huge matrix use randomized or truncated (Lanczos) SVD instead of decomposing everything.

In interviews

The SVD's headline facts are the targets: it exists for any matrix, $U$ and $V$ are orthogonal, singular values are non-negative and equal $\sqrt{\text{eigenvalues of } A^\top A}$. Expect "how is the SVD related to PCA?", PCA is the SVD (or eigen-decomposition of the covariance) of the centered data, principal components are the right singular vectors, explained variance is $\sigma_i^2$. The Eckart–Young low-rank result is a strong thing to know by name: truncating the SVD is the optimal low-rank approximation, which grounds denoising and compression. Be ready to explain the condition number as $\sigma_1/\sigma_r$ and why tiny singular values make inversion dangerous, motivating the pseudo-inverse and truncated SVD as regularization. Tie it to Positive Semidefinite Matrices (where $A^\top A$ lives) and Eigenvalues & Eigenvectors to show you see the whole spectral picture.