---

Suppose that we would like to estimate the value of an unobserved random variable $X$, given that we have observed $Y=y$. In general, our estimate $\hat{x}$ is a function of $y$ \begin{align} \hat{x}=g(y). \end{align} For example, the MMSE estimate of $X$ given $Y=y$ is \begin{align} g(y)=E[X|Y=y]. \end{align} We might face some difficulties if we want to use the MMSE in practice. First, the function $g(y)=E[X|Y=y]$ might have a complicated form. Specifically, if $X$ and $Y$ are random vectors, computing $E[X|Y=y]$ might not be easy. Moreover, to find $E[X|Y=y]$ we need to know $f_{X|Y}(y)$, which might not be easy to find in some problems. To address these issues, we might want to use a simpler function $g(y)$ to estimate $X$. In particular, we might want $g(y)$ to be a linear function of $y$.

Suppose that we would like to have an estimator for $X$ of the form

\begin{align} \hat{X}_L=g(Y)=aY+b, \end{align} where $a$ and $b$ are some real numbers to be determined. More specifically, our goal is to choose $a$ and $b$ such that the MSE of the above estimator \begin{align} MSE=E[(X-\hat{X}_L)^2] \end{align} is minimized. We call the resulting estimator the linear MMSE estimator. The following theorem gives us the optimal values for $a$ and $b$.

---

Theorem
Let $X$ and $Y$ be two random variables with finite means and variances. Also, let $\rho$ be the correlation coefficient of $X$ and $Y$. Consider the function \begin{align} h(a,b)=E[(X-aY-b)^2]. \end{align} Then,

1. The function $h(a,b)$ is minimized if \begin{align} a=a^*=\frac{\textrm{Cov}(X,Y)}{\textrm{Var}(Y)}, \quad b=b^*=EX-aEY. \end{align}
2. We have $h(a^*,b^*)=(1-\rho^2)\textrm{Var}(X)$.
3. $E[(X-a^*Y-b^*)Y]=0$ (orthogonality principle).

Proof: We have \begin{align} h(a,b)&=E[(X-aY-b)^2]\\ &=E[X^2+a^2Y^2+b^2-2aXY-2bX+2abY]\\ &=EX^2+a^2EY^2+b^2-2aEXY-2bEX+2abEY. \end{align} Thus, $h(a,b)$ is a quadratic function of $a$ and $b$. We take the derivatives with respect to $a$ and $b$ and set them to zero, so we obtain \begin{align} EY^2 \cdot a +EY \cdot b= EXY \hspace{30pt} (9.4) \end{align} \begin{align} EY \cdot a+ b=EX \hspace{30pt} (9.5) \end{align} Solving for $a$ and $b$, we obtain \begin{align} a^*=\frac{\textrm{Cov}(X,Y)}{\textrm{Var}(Y)}, \quad b^*=EX-aEY. \end{align} It can be verified that the above values do in fact minimize $h(a,b)$. Note that Equation 9.5 implies that $E[X-a^*Y-b^*]=0$. Therefore, \begin{align} h(a^*,b^*)&=E[(X-a^*Y-b^*)^2]\\ &=\textrm{Var}(X-a^*Y-b^*)\\ &=\textrm{Var}(X-a^*Y)\\ &=\textrm{Var}(X)+a^{*2}\textrm{Var}(Y)-2 a^* \textrm{Cov}(X,Y)\\ &=\textrm{Var}(X)+\frac{\textrm{Cov}(X,Y)^2}{\textrm{Var}(Y)^2}\textrm{Var}(Y)-2 \frac{\textrm{Cov}(X,Y)}{\textrm{Var}(Y)} \textrm{Cov}(X,Y)\\ &=\textrm{Var}(X)-\frac{\textrm{Cov}(X,Y)^2}{\textrm{Var}(Y)}\\ &=(1-\rho^2)\textrm{Var}(X). \end{align} Finally, note that \begin{align} E[(X-a^*Y-b^*)Y]&=EXY-a^*EY^2-b^*EY\\ &=0 \quad (\textrm{by Equation 9.4}). \end{align}

---

Note that $\tilde{X}=X-a^*Y-b^*$ is the error in the linear MMSE estimation of $X$ given $Y$. From the above theorem, we conclude that \begin{align} &E[\tilde{X}]=0,\\ &E[\tilde{X} Y]=0. \end{align} In sum, we can write the linear MMSE estimator of $X$ given $Y$ as \begin{align} \hat{X}_L=\frac{\textrm{Cov}(X,Y)}{\textrm{Var}(Y)} (Y-EY)+ EX. \end{align} If $\rho=\rho(X,Y)$ is the correlation coefficient of $X$ and $Y$, then $\textrm{Cov}(X,Y)=\rho \sigma_X \sigma_Y$, so the above formula can be written as \begin{align} \hat{X}_L=\frac{\rho \sigma_X}{\sigma_Y} (Y-EY)+ EX. \end{align} Note that to compute the linear MMSE estimates, we only need to know expected values, variances, and the covariance. Let us look at an example.

---

Example
Suppose $X \sim Uniform(1,2)$, and given $X=x$, $Y$ is exponential with parameter $\lambda=\frac{1}{x}$.

1. Find the linear MMSE estimate of $X$ given $Y$.
2. Find the MSE of this estimator.
3. Check that $E[\tilde{X} Y]=0$.

---