The best shrinkage factor for a noisy estimate

Question

You observe X \sim \mathcal{N}(	heta, \sigma^2) and estimate 	heta with a scaled version \hat{	heta} = cX for some constant c. Which c minimizes the mean squared error, and why is it always less than 1?

Accepted Answer

Write MSE as variance plus bias squared. With \hat{	heta} = cX: - \operatorname{Var}(cX) = c^2 \sigma^2. - \mathbb{E}[cX] = c	heta, so the bias is c	heta - 	heta = (c - 1)	heta, and bias^2 = (c-1)^2 	heta^2. \operatorname{MSE}(c) = c^2 \sigma^2 + (c - 1)^2 	heta^2. Minimize. Differentiate and set to zero: \frac{d}{dc}\operatorname{MSE} = 2c\sigma^2 + 2(c-1)	heta^2 = 0 \;\Rightarrow\; c(\sigma^2 + 	heta^2) = 	heta^2 \;\Rightarrow\; c^ = \frac{	heta^2}{\sigma^2 + 	heta^2}. Why c^ < 1. The denominator \sigma^2 + 	heta^2 exceeds the numerator 	heta^2 whenever there is any noise (\si…

The best shrinkage factor for a noisy estimate

Your answer

More Statistics questions