8.2.5 Solved Problems
Let $X$ be the height of a randomly chosen individual from a population. In order to estimate the mean and variance of $X$, we observe a random sample $X_1$,$X_2$,$\cdots$,$X_7$. Thus, $X_i$'s are i.i.d. and have the same distribution as $X$. We obtain the following values (in centimeters):
\begin{equation}%\label{} 166.8, 171.4, 169.1, 178.5, 168.0, 157.9, 170.1 \end{equation}Find the values of the sample mean, the sample variance, and the sample standard deviation for the observed sample.
- Solution
-
\begin{align}%\label{}
\overline{X}&=\frac{X_1+X_2+X_3+X_4+X_5+X_6+X_7}{7}\\
&=\frac{166.8+171.4+169.1+178.5+168.0+157.9+170.1}{7}\\
&=168.8
\end{align}
The sample variance is given by
\begin{align}%\label{}
{S}^2=\frac{1}{7-1} \sum_{k=1}^7 (X_k-168.8)^2&=37.7
\end{align}
Finally, the sample standard deviation is given by
\begin{align}%\label{}
&= \sqrt{S^2}=6.1
\end{align}
The following MATLAB code can be used to obtain these values:
x=[166.8, 171.4, 169.1, 178.5, 168.0, 157.9, 170.1];
m=mean(x);
v=var(x);
s=std(x);
-
\begin{align}%\label{}
\overline{X}&=\frac{X_1+X_2+X_3+X_4+X_5+X_6+X_7}{7}\\
&=\frac{166.8+171.4+169.1+178.5+168.0+157.9+170.1}{7}\\
&=168.8
\end{align}
The sample variance is given by
\begin{align}%\label{}
{S}^2=\frac{1}{7-1} \sum_{k=1}^7 (X_k-168.8)^2&=37.7
\end{align}
Finally, the sample standard deviation is given by
\begin{align}%\label{}
&= \sqrt{S^2}=6.1
\end{align}
The following MATLAB code can be used to obtain these values:
Problem
Prove the following:
If $\hat{\Theta}_1$ is an unbiased estimator for $\theta$, and $W$ is a zero mean random variable, then
\begin{align}%\label{} \hat{\Theta}_2=\hat{\Theta}_1+W \end{align} is also an unbiased estimator for $\theta$.- If $\hat{\Theta}_1$ is an estimator for $\theta$ such that $E[\hat{\Theta}_1]=a \theta+b$, where $a \neq 0$, show that
\begin{align}%\label{}
\hat{\Theta}_2=\frac{\hat{\Theta}_1-b}{a}
\end{align}
is an unbiased estimator for $\theta$.
- Solution
-
- We have \begin{align} E[\hat{\Theta}_2]&=E[\hat{\Theta}_1]+E[W] & (\textrm{by linearity of expectation})\\ &=\theta+0 & (\textrm{since $\hat{\Theta}_1$ is unbiased and } EW=0)\\ &=\theta. \end{align} Thus, $\hat{\Theta}_2$ is an unbiased estimator for $\theta$.
- We have \begin{align}%\label{} E[\hat{\Theta}_2]&=\frac{E[\hat{\Theta}_1]-b}{a} (\textrm{by linearity of expectation})\\ &=\frac{a \theta+b-b}{a}\\ &=\theta. \end{align} Thus, $\hat{\Theta}_2$ is an unbiased estimator for $\theta$.
-
Problem
Let $X_1$, $X_2$, $X_3$, $...$, $X_n$ be a random sample from a $Uniform(0,\theta)$ distribution, where $\theta$ is unknown. Define the estimator
\begin{align} \hat{\Theta}_n=\max \{X_1, X_2, \cdots, X_n \}. \end{align}- Find the bias of $\hat{\Theta}_n$, $B(\hat{\Theta}_n)$.
- Find the MSE of $\hat{\Theta}_n$, $MSE(\hat{\Theta}_n)$.
- Is $\hat{\Theta}_n$ a consistent estimator of $\theta$?
- Solution
-
If $X \sim Uniform (0, \theta)$, then the PDF and CDF of $X$ are given by
\begin{align}
\nonumber f_X(x) = \left\{
\begin{array}{l l}
\frac{1}{\theta} & \quad 0 \leq x \leq \theta \\
& \quad \\
0 & \quad \text{otherwise}
\end{array} \right.
\end{align}
and
\begin{align}
\nonumber F_X(x) = \left\{
\begin{array}{l l}
0 & \quad x<0 \\
& \quad \\
\frac{x}{\theta} & \quad 0 \leq x \leq \theta \\
& \quad \\
1 & \quad x>1
\end{array} \right.
\end{align}
By Theorem 8.1, the PDF of $\hat{\Theta}_n$ is given by
\begin{align}%\label{}
f_{\hat{\Theta}_n}(y)&=n f_X(x) \big[ F_X(x)\big]^{n-1}\\
&=\left\{
\begin{array}{l l}
\frac{ny^{n-1}}{\theta^n} & \quad 0 \leq y \leq \theta \\
& \quad \\
0 & \quad \text{otherwise}
\end{array} \right.
\end{align}
- To find the bias of $\hat{\Theta}_n$, we have \begin{align} E[\hat{\Theta}_n]&= \int_{0}^{\theta} y \cdot \frac{ny^{n-1}}{\theta^n} dy \\ &=\frac{n}{n+1} \theta. \end{align} Thus, the bias is given by \begin{align} B(\hat{\Theta}_n)&=E[\hat{\Theta}_n]-\theta \\ &= \frac{n}{n+1} \theta-\theta\\ &= -\frac{\theta}{n+1}. \end{align}
- To find $MSE(\hat{\Theta}_n)$, we can write \begin{align}%\label{} MSE(\hat{\Theta}_n)&=\textrm{Var}(\hat{\Theta}_n)+B(\hat{\Theta}_n)^2\\ &=\textrm{Var}(\hat{\Theta}_n)+ \frac{\theta^2}{(n+1)^2}. \end{align} Thus, we need to find $\textrm{Var}(\hat{\Theta})$. We have \begin{align} E\left[\hat{\Theta}_n^2\right]&= \int_{0}^{\theta} y^2 \cdot \frac{ny^{n-1}}{\theta^n} dy \\ &=\frac{n}{n+2} \theta^2. \end{align} Thus, \begin{align} \textrm{Var}(\hat{\Theta}_n)&=E\left[\hat{\Theta}_n^2\right]- \big(E[\hat{\Theta}_n]\big)^2\\ &=\frac{n}{(n+2)(n+1)^2} \theta^2. \end{align} Therefore, \begin{align}%\label{} MSE(\hat{\Theta}_n)&=\frac{n}{(n+2)(n+1)^2} \theta^2+ \frac{\theta^2}{(n+1)^2}\\ &=\frac{2\theta^2}{(n+2)(n+1)}. \end{align}
- Note that \begin{align}%\label{eq:union-bound} \lim_{n \rightarrow \infty} MSE(\hat{\Theta}_n)=\lim_{n \rightarrow \infty} \frac{2\theta^2}{(n+2)(n+1)}=0. \end{align} Thus, by Theorem 8.2, $\hat{\Theta}_n$ is a consistent estimator of $\theta$.
-
If $X \sim Uniform (0, \theta)$, then the PDF and CDF of $X$ are given by
\begin{align}
\nonumber f_X(x) = \left\{
\begin{array}{l l}
\frac{1}{\theta} & \quad 0 \leq x \leq \theta \\
& \quad \\
0 & \quad \text{otherwise}
\end{array} \right.
\end{align}
and
\begin{align}
\nonumber F_X(x) = \left\{
\begin{array}{l l}
0 & \quad x<0 \\
& \quad \\
\frac{x}{\theta} & \quad 0 \leq x \leq \theta \\
& \quad \\
1 & \quad x>1
\end{array} \right.
\end{align}
By Theorem 8.1, the PDF of $\hat{\Theta}_n$ is given by
\begin{align}%\label{}
f_{\hat{\Theta}_n}(y)&=n f_X(x) \big[ F_X(x)\big]^{n-1}\\
&=\left\{
\begin{array}{l l}
\frac{ny^{n-1}}{\theta^n} & \quad 0 \leq y \leq \theta \\
& \quad \\
0 & \quad \text{otherwise}
\end{array} \right.
\end{align}
Problem
Let $X_1$, $X_2$, $X_3$, $...$, $X_n$ be a random sample from a $Geometric(\theta)$ distribution, where $\theta$ is unknown. Find the maximum likelihood estimator (MLE) of $\theta$ based on this random sample.
- Solution
- If $X_i \sim Geometric(\theta)$, then \begin{align} P_{X_i}(x;\theta) = (1-\theta)^{x-1} \theta. \end{align} Thus, the likelihood function is given by \begin{align} L(x_1, x_2, \cdots, x_n; \theta)&=P_{X_1 X_2 \cdots X_n}(x_1, x_2, \cdots, x_n; \theta)\\ &=P_{X_1}(x_1;\theta) P_{X_2}(x_2;\theta) \cdots P_{X_n}(x_n;\theta)\\ &=(1-\theta)^{\left[\sum_{i=1}^n x_i-n\right]} \theta^{n}. \end{align} Then, the log likelihood function is given by \begin{align} \ln L(x_1, x_2, \cdots, x_n; \theta)= \bigg({\sum_{i=1}^n x_i-n} \bigg) \ln (1-\theta)+ n \ln {\theta}. \end{align} Thus, \begin{align} \frac{d \ln L(x_1, x_2, \cdots, x_n; \theta)}{d\theta}= \bigg({\sum_{i=1}^n x_i-n} \bigg) \cdot \frac{-1}{1-\theta}+ \frac{n} {\theta}. \end{align} By setting the derivative to zero, we can check that the maximizing value of $\theta$ is given by \begin{align} \hat{\theta}_{ML}= \frac{n} {\sum_{i=1}^n x_i}. \end{align} Thus, the MLE can be written as \begin{align} \hat{\Theta}_{ML}= \frac{n} {\sum_{i=1}^n X_i}. \end{align}
Problem
Let $X_1$, $X_2$, $X_3$, $...$, $X_n$ be a random sample from a $Uniform(0,\theta)$ distribution, where $\theta$ is unknown. Find the maximum likelihood estimator (MLE) of $\theta$ based on this random sample.
- Solution
- If $X_i \sim Uniform(0,\theta)$, then \begin{align} \nonumber f_X(x) = \left\{ \begin{array}{l l} \frac{1}{\theta} & \quad 0 \leq x \leq \theta \\ & \quad \\ 0 & \quad \text{otherwise} \end{array} \right. \end{align} The likelihood function is given by \begin{align} L(x_1, x_2, \cdots, x_n; \theta)&=f_{X_1 X_2 \cdots X_n}(x_1, x_2, \cdots, x_n; \theta)\\ &=f_{X_1}(x_1;\theta) f_{X_2}(x_2;\theta) \cdots f_{X_n}(x_n;\theta)\\ &=\left\{ \begin{array}{l l} \frac{1}{\theta^n} & \quad 0 \leq x_1, x_2, \cdots, x_n \leq \theta \\ & \quad \\ 0 & \quad \text{otherwise} \end{array} \right. \end{align} Note that $\frac{1}{\theta^n}$ is a decreasing function of $\theta$. Thus, to maximize it, we need to choose the smallest possible value for $\theta$. For $i=1,2,...,n$, we need to have $\theta \geq x_i$. Thus, the smallest possible value for $\theta$ is \begin{align} \hat{\theta}_{ML}= \max(x_1,x_2, \cdots, x_n). \end{align} Therefore, the MLE can be written as \begin{align} \hat{\Theta}_{ML}= \max(X_1,X_2, \cdots, X_n). \end{align} Note that this is one of those cases wherein $\hat{\theta}_{ML}$ cannot be obtained by setting the derivative of the likelihood function to zero. Here, the maximum is achieved at an endpoint of the acceptable interval.
