8.2.2 Point Estimators for Mean and Variance

The above discussion suggests the sample mean, $\overline{X}$, is often a reasonable point estimator for the mean. Now, suppose that we would like to estimate the variance of a distribution $\sigma^2$. Assuming $0<\sigma^2<\infty$, by definition \begin{align}%\label{} \sigma^2=E[(X-\mu)^2]. \end{align} Thus, the variance itself is the mean of the random variable $Y=(X-\mu)^2$. This suggests the following estimator for the variance \begin{align}%\label{} \hat{\sigma}^2=\frac{1}{n} \sum_{k=1}^n (X_k-\mu)^2. \end{align} By linearity of expectation, $\hat{\sigma}^2$ is an unbiased estimator of $\sigma^2$. Also, by the weak law of large numbers, $\hat{\sigma}^2$ is also a consistent estimator of $\sigma^2$. However, in practice we often do not know the value of $\mu$. Thus, we may replace $\mu$ by our estimate of the $\mu$, the sample mean, to obtain the following estimator for $\sigma^2$: \begin{align}%\label{} {\overline{S}}^2=\frac{1}{n} \sum_{k=1}^n (X_k-\overline{X})^2. \end{align} Using a little algebra, you can show that \begin{align}%\label{} {\overline{S}}^2=\frac{1}{n} \left(\sum_{k=1}^n X^2_k-n\overline{X}^2\right). \end{align}

Let $X_1$, $X_2$, $X_3$, $...$, $X_n$ be a random sample with mean $EX_i=\mu$, and variance $\mathrm{Var}(X_i)=\sigma^2$. Suppose that we use

\begin{align}%\label{} {\overline{S}}^2=\frac{1}{n} \sum_{k=1}^n (X_k-\overline{X})^2=\frac{1}{n} \left(\sum_{k=1}^n X^2_k-n\overline{X}^2\right) \end{align} to estimate $\sigma^2$. Find the bias of this estimator \begin{align}%\label{} B({\overline{S}}^2)=E[{\overline{S}}^2]-\sigma^2. \end{align}
  • Solution
    • First note that \begin{align}%\label{} E{\overline{X}}^2 &=\big(E\overline{X})^2+\mathrm{Var}(\overline{X})\\ &=\mu^2+\frac{\sigma^2}{n}. \end{align} Thus, \begin{align}%\label{} E[{\overline{S}}^2]&=\frac{1}{n} \left(\sum_{k=1}^n EX^2_k-nE\overline{X}^2\right)\\ &=\frac{1}{n} \left(n(\mu^2+\sigma^2)-n\left(\mu^2+\frac{\sigma^2}{n}\right)\right)\\ &=\frac{n-1}{n}\sigma^2. \end{align} Therefore, \begin{align}%\label{} B({\overline{S}}^2)&=E[{\overline{S}}^2]-\sigma^2\\ &=-\frac{\sigma^2}{n}. \end{align}

We conclude that $\overline{S}^2$ is a biased estimator of the variance. Nevertheless, note that if $n$ is relatively large, the bias is very small. Since $E[{\overline{S}}^2]=\frac{n-1}{n}\sigma^2$, we can obtain an unbiased estimator of $\sigma^2$ by multiplying $\overline{S}^2$ by $\frac{n}{n-1}$. Thus, we define \begin{align}%\label{} {S}^2=\frac{1}{n-1} \sum_{k=1}^n (X_k-\overline{X})^2=\frac{1}{n-1} \left(\sum_{k=1}^n X^2_k-n\overline{X}^2\right). \end{align} By the above discussion, $S^2$ is an unbiased estimator of the variance. We call it the sample variance. We should note that if $n$ is large, the difference between $S^2$ and $\overline{S}^2$ is very small. We also define the sample standard deviation as \begin{align}%\label{} S&= \sqrt{S^2}. \end{align} Although the sample standard deviation is usually used as an estimator for the standard deviation, it is a biased estimator. To see this, note that $S$ is random, so $\mathrm{Var}(S)>0$. Thus, \begin{align}%\label{} 0<\mathrm{Var}(S)&= ES^2-(ES)^2\\ &=\sigma^2-(ES)^2. \end{align} Therefore, $ES<\sigma$, which means that $S$ is a biased estimator of $\sigma$.
Let $X_1$, $X_2$, $X_3$, $...$, $X_n$ be a random sample with mean $EX_i=\mu<\infty$, and variance $0<\mathrm{Var}(X_i)=\sigma^2<\infty$. The sample variance of this random sample is defined as \begin{align}%\label{} {S}^2=\frac{1}{n-1} \sum_{k=1}^n (X_k-\overline{X})^2=\frac{1}{n-1} \left(\sum_{k=1}^n X^2_k-n\overline{X}^2\right). \end{align} The sample variance is an unbiased estimator of $\sigma^2$. The sample standard deviation is defined as \begin{align}%\label{} S&= \sqrt{S^2}, \end{align} and is commonly used as an estimator for $\sigma$. Nevertheless, $S$ is a biased estimator of $\sigma$.

You can use the mean command in MATLAB to compute the sample mean for a given sample. More specifically, for a given vector $x=$$[$$x_1$, $x_2$, $\cdots$, $x_n$ $]$, mean(x) returns the sample average \begin{align}%\label{} \frac{x_1+x_2+\cdots+x_n}{n}. \end{align} Also, the functions var and std can be used to compute the sample variance and the sample standard deviation respectively.

Let $T$ be the time that is needed for a specific task in a factory to be completed. In order to estimate the mean and variance of $T$, we observe a random sample $T_1$,$T_2$,$\cdots$,$T_6$. Thus, $T_i$'s are i.i.d. and have the same distribution as $T$. We obtain the following values (in minutes):

\begin{equation}%\label{} 18, 21, 17, 16, 24, 20. \end{equation} Find the values of the sample mean, the sample variance, and the sample standard deviation for the observed sample.
  • Solution
    • The sample mean is \begin{align}%\label{} \overline{T}&=\frac{T_1+T_2+T_3+T_4+T_5+T_6}{6}\\ &=\frac{18+21+17+16+24+20}{6}\\ &=19.33 \end{align} The sample variance is given by \begin{align}%\label{} {S}^2=\frac{1}{6-1} \sum_{k=1}^6 (T_k-19.333)^2&=8.67 \end{align} Finally, the sample standard deviation is given by \begin{align}%\label{} S&= \sqrt{S^2}=2.94 \end{align}

You can use the following MATLAB code to compute the above values:
t = [18, 21, 17, 16, 24, 20];
m = mean(t);
v = var(t);
s = std(t);

The print version of the book is available through Amazon here.

Book Cover