Maximum Likelihood Estimation (MLE)

Maximum likelihood inverts the usual probabilistic question. Instead of asking "given the parameters, how probable is the data," it asks "given the data I actually observed, which parameters make that data most probable." That inversion turns estimation into optimization, and it comes with a package of asymptotic guarantees, consistency, efficiency, and normality, that no other general recipe matches. GARCH, logit, and most of the volatility and default models a desk runs are fit by maximum likelihood.

The likelihood and log-likelihood

Given data $x_1, \dots, x_n$ drawn from a density $f(x;\theta)$, the likelihood treats the data as fixed and $\theta$ as the variable:

$L(\theta) = \prod_{i=1}^n f(x_i; \theta).$

Because products are numerically nasty and analytically awkward, we work with the log-likelihood $\ell(\theta) = \log L(\theta) = \sum_i \log f(x_i;\theta)$, which is maximized at the same $\hat\theta$ since $\log$ is monotone. The MLE is

$\hat\theta_{\text{MLE}} = \arg\max_\theta \ \ell(\theta).$

At an interior maximum the gradient, the score, is zero:

$s(\theta) = \frac{\partial \ell}{\partial \theta} = \sum_i \frac{\partial \log f(x_i;\theta)}{\partial \theta} = 0.$

This score equation is the workhorse. A key identity is that the score has mean zero at the true parameter, $\mathbb{E}_\theta[s(\theta)] = 0$, which is what makes the MLE consistent.

Fisher information and the curvature of the log-likelihood

How sharply the log-likelihood peaks around $\hat\theta$ measures how much the data tell you. The Fisher information is the variance of the score, equivalently the expected curvature:

$I(\theta) = \operatorname{Var}\big(s(\theta)\big) = -\,\mathbb{E}\!\left[\frac{\partial^2 \ell}{\partial \theta^2}\right].$

A steeply curved (high-information) log-likelihood localizes $\theta$ tightly; a flat one barely constrains it. This is the same $I(\theta)$ that sets the Cramér–Rao floor in Bias, Variance, and the Quality of Estimators, and the reason MLE is the benchmark is that it achieves that floor asymptotically.

Asymptotic normality

The central result. Under regularity conditions (smoothness, identifiability, an interior true value), the MLE is consistent and

$\sqrt{n}\,(\hat\theta - \theta_0) \ \xrightarrow{d}\ \mathcal{N}\!\big(0,\ I_1(\theta_0)^{-1}\big),$

where $I_1$ is the per-observation information. A one-line sketch: Taylor-expand the score around $\theta_0$, $0 = s(\hat\theta) \approx s(\theta_0) + s'(\theta_0)(\hat\theta - \theta_0)$, solve $\hat\theta - \theta_0 \approx -s(\theta_0)/s'(\theta_0)$. The numerator is a sum of mean-zero terms (a The Central Limit Theorem object) and the denominator converges to $-I(\theta_0)$ by the The Law of Large Numbers, and the ratio is asymptotically Gaussian with variance $I_1^{-1}$. In practice you estimate the covariance by the inverse observed information $\big(-\partial^2\ell/\partial\theta\partial\theta^\top\big)^{-1}$ at $\hat\theta$, this is where the standard errors printed next to every GARCH coefficient come from.

The invariance property

MLE has a property no other estimator shares so cleanly: if $\hat\theta$ is the MLE of $\theta$, then for any function $g$, the MLE of $g(\theta)$ is simply $g(\hat\theta)$. You do not re-optimize. Estimate a variance by MLE and the MLE of the standard deviation is its square root; estimate log-odds and the MLE of the probability is the logistic transform. Invariance is why practitioners parameterize models however is numerically convenient (e.g. estimating $\log\sigma$ to keep it positive) and transform back without penalty.

Worked example: MLE for the normal

Let $x_1, \dots, x_n \sim \mathcal{N}(\mu, \sigma^2)$ i.i.d. The log-likelihood is

$\ell(\mu, \sigma^2) = -\frac{n}{2}\log(2\pi) - \frac{n}{2}\log\sigma^2 - \frac{1}{2\sigma^2}\sum_{i}(x_i - \mu)^2.$

Setting $\partial\ell/\partial\mu = \frac{1}{\sigma^2}\sum_i (x_i - \mu) = 0$ gives $\hat\mu = \bar x$. Setting $\partial\ell/\partial\sigma^2 = -\frac{n}{2\sigma^2} + \frac{1}{2\sigma^4}\sum_i(x_i-\mu)^2 = 0$ gives

$\hat\sigma^2_{\text{MLE}} = \frac{1}{n}\sum_i (x_i - \bar x)^2.$

Note the divisor $n$, not $n-1$: the MLE of the variance is biased downward, exactly the estimator from the bias–variance discussion. MLE optimizes fit, not unbiasedness, a clean reminder that its guarantees are asymptotic. Under normality, minimizing $\sum(x_i - x_i^\top\beta)^2$ is the MLE for a regression, which is why OLS and Gaussian MLE coincide (see Ordinary Least Squares (OLS)).

Failure modes in financial data

• Misspecified likelihood. MLE is only as good as the density you assume. Fitting a Gaussian likelihood to fat-tailed returns underweights tail events; the estimator becomes quasi-MLE, still often consistent for the mean/variance but with wrong standard errors unless you use a sandwich (robust) covariance. Many desks fit GARCH with a Student-$t$ innovation for exactly this reason. See GARCH Volatility Models.
• Flat or multimodal likelihoods. Weakly identified parameters (e.g. a near-zero GARCH mean-reversion term, or regime-switching models) produce ridges and multiple local maxima; the optimizer's answer depends on the starting value. Always try multiple starts.
• Boundary solutions. Variance and persistence parameters live on constrained sets. The MLE can pile up on a boundary (an IGARCH-like unit persistence, a correlation hitting $\pm 1$), where the standard asymptotic-normality theory does not apply and the usual t-statistics are invalid.
• Small samples. All the guarantees are asymptotic. With a few hundred daily observations and heavy tails, the finite-sample bias and the gap from the Cramér–Rao bound can both be large; bootstrap the standard errors rather than trusting the information-matrix ones. See Bootstrap and Resampling.

In interviews

Be able to write the log-likelihood for a simple model, take the score, and solve, the normal case ($\hat\mu = \bar x$, $\hat\sigma^2 = \frac1n\sum(x_i-\bar x)^2$) is the canonical warm-up, and mentioning that the variance MLE uses $n$ not $n-1$ signals you understand the bias. State the three asymptotic properties (consistent, efficient, asymptotically normal with variance $I^{-1}$) and know that MLE attains the Cramér–Rao bound. Invariance is a favorite quick question: "what's the MLE of $\sigma$ if you have the MLE of $\sigma^2$?", the square root, no reoptimization. A deeper probe connects MLE to Bayesian estimation: the MLE is the posterior mode under a flat prior, the bridge to Bayesian Inference.