8.2.5 Solved Problems

Problem

Let $X$ be the height of a randomly chosen individual from a population. In order to estimate the mean and variance of $X$, we observe a random sample $X_1$,$X_2$,$\cdots$,$X_7$. Thus, $X_i$'s are i.i.d. and have the same distribution as $X$. We obtain the following values (in centimeters):

\begin{equation}%\label{} 166.8, 171.4, 169.1, 178.5, 168.0, 157.9, 170.1 \end{equation}

Find the values of the sample mean, the sample variance, and the sample standard deviation for the observed sample.

  • Solution
    • \begin{align}%\label{} \overline{X}&=\frac{X_1+X_2+X_3+X_4+X_5+X_6+X_7}{7}\\ &=\frac{166.8+171.4+169.1+178.5+168.0+157.9+170.1}{7}\\ &=168.8 \end{align} The sample variance is given by \begin{align}%\label{} {S}^2=\frac{1}{7-1} \sum_{k=1}^7 (X_k-168.8)^2&=37.7 \end{align} Finally, the sample standard deviation is given by \begin{align}%\label{} &= \sqrt{S^2}=6.1 \end{align} The following MATLAB code can be used to obtain these values:
      x=[166.8, 171.4, 169.1, 178.5, 168.0, 157.9, 170.1];
      m=mean(x);
      v=var(x);
      s=std(x);


Problem

Prove the following:

  1. If $\hat{\Theta}_1$ is an unbiased estimator for $\theta$, and $W$ is a zero mean random variable, then

    \begin{align}%\label{} \hat{\Theta}_2=\hat{\Theta}_1+W \end{align} is also an unbiased estimator for $\theta$.
  2. If $\hat{\Theta}_1$ is an estimator for $\theta$ such that $E[\hat{\Theta}_1]=a \theta+b$, where $a \neq 0$, show that \begin{align}%\label{} \hat{\Theta}_2=\frac{\hat{\Theta}_1-b}{a} \end{align}

    is an unbiased estimator for $\theta$.

  • Solution
      1. We have \begin{align} E[\hat{\Theta}_2]&=E[\hat{\Theta}_1]+E[W] & (\textrm{by linearity of expectation})\\ &=\theta+0 & (\textrm{since $\hat{\Theta}_1$ is unbiased and } EW=0)\\ &=\theta. \end{align} Thus, $\hat{\Theta}_2$ is an unbiased estimator for $\theta$.
      2. We have \begin{align}%\label{} E[\hat{\Theta}_2]&=\frac{E[\hat{\Theta}_1]-b}{a} (\textrm{by linearity of expectation})\\ &=\frac{a \theta+b-b}{a}\\ &=\theta. \end{align} Thus, $\hat{\Theta}_2$ is an unbiased estimator for $\theta$.


Problem

Let $X_1$, $X_2$, $X_3$, $...$, $X_n$ be a random sample from a $Uniform(0,\theta)$ distribution, where $\theta$ is unknown. Define the estimator

\begin{align} \hat{\Theta}_n=\max \{X_1, X_2, \cdots, X_n \}. \end{align}
  1. Find the bias of $\hat{\Theta}_n$, $B(\hat{\Theta}_n)$.
  2. Find the MSE of $\hat{\Theta}_n$, $MSE(\hat{\Theta}_n)$.
  3. Is $\hat{\Theta}_n$ a consistent estimator of $\theta$?
  • Solution
    • If $X \sim Uniform (0, \theta)$, then the PDF and CDF of $X$ are given by \begin{align} \nonumber f_X(x) = \left\{ \begin{array}{l l} \frac{1}{\theta} & \quad 0 \leq x \leq \theta \\ & \quad \\ 0 & \quad \text{otherwise} \end{array} \right. \end{align} and \begin{align} \nonumber F_X(x) = \left\{ \begin{array}{l l} 0 & \quad x<0 \\ & \quad \\ \frac{x}{\theta} & \quad 0 \leq x \leq \theta \\ & \quad \\ 1 & \quad x>1 \end{array} \right. \end{align} By Theorem 8.1, the PDF of $\hat{\Theta}_n$ is given by \begin{align}%\label{} f_{\hat{\Theta}_n}(y)&=n f_X(x) \big[ F_X(x)\big]^{n-1}\\ &=\left\{ \begin{array}{l l} \frac{ny^{n-1}}{\theta^n} & \quad 0 \leq y \leq \theta \\ & \quad \\ 0 & \quad \text{otherwise} \end{array} \right. \end{align}
      1. To find the bias of $\hat{\Theta}_n$, we have \begin{align} E[\hat{\Theta}_n]&= \int_{0}^{\theta} y \cdot \frac{ny^{n-1}}{\theta^n} dy \\ &=\frac{n}{n+1} \theta. \end{align} Thus, the bias is given by \begin{align} B(\hat{\Theta}_n)&=E[\hat{\Theta}_n]-\theta \\ &= \frac{n}{n+1} \theta-\theta\\ &= -\frac{\theta}{n+1}. \end{align}
      2. To find $MSE(\hat{\Theta}_n)$, we can write \begin{align}%\label{} MSE(\hat{\Theta}_n)&=\textrm{Var}(\hat{\Theta}_n)+B(\hat{\Theta}_n)^2\\ &=\textrm{Var}(\hat{\Theta}_n)+ \frac{\theta^2}{(n+1)^2}. \end{align} Thus, we need to find $\textrm{Var}(\hat{\Theta})$. We have \begin{align} E\left[\hat{\Theta}_n^2\right]&= \int_{0}^{\theta} y^2 \cdot \frac{ny^{n-1}}{\theta^n} dy \\ &=\frac{n}{n+2} \theta^2. \end{align} Thus, \begin{align} \textrm{Var}(\hat{\Theta}_n)&=E\left[\hat{\Theta}_n^2\right]- \big(E[\hat{\Theta}_n]\big)^2\\ &=\frac{n}{(n+2)(n+1)^2} \theta^2. \end{align} Therefore, \begin{align}%\label{} MSE(\hat{\Theta}_n)&=\frac{n}{(n+2)(n+1)^2} \theta^2+ \frac{\theta^2}{(n+1)^2}\\ &=\frac{2\theta^2}{(n+2)(n+1)}. \end{align}
      3. Note that \begin{align}%\label{eq:union-bound} \lim_{n \rightarrow \infty} MSE(\hat{\Theta}_n)=\lim_{n \rightarrow \infty} \frac{2\theta^2}{(n+2)(n+1)}=0. \end{align} Thus, by Theorem 8.2, $\hat{\Theta}_n$ is a consistent estimator of $\theta$.


Problem

Let $X_1$, $X_2$, $X_3$, $...$, $X_n$ be a random sample from a $Geometric(\theta)$ distribution, where $\theta$ is unknown. Find the maximum likelihood estimator (MLE) of $\theta$ based on this random sample.

  • Solution
    • If $X_i \sim Geometric(\theta)$, then \begin{align} P_{X_i}(x;\theta) = (1-\theta)^{x-1} \theta. \end{align} Thus, the likelihood function is given by \begin{align} L(x_1, x_2, \cdots, x_n; \theta)&=P_{X_1 X_2 \cdots X_n}(x_1, x_2, \cdots, x_n; \theta)\\ &=P_{X_1}(x_1;\theta) P_{X_2}(x_2;\theta) \cdots P_{X_n}(x_n;\theta)\\ &=(1-\theta)^{\left[\sum_{i=1}^n x_i-n\right]} \theta^{n}. \end{align} Then, the log likelihood function is given by \begin{align} \ln L(x_1, x_2, \cdots, x_n; \theta)= \bigg({\sum_{i=1}^n x_i-n} \bigg) \ln (1-\theta)+ n \ln {\theta}. \end{align} Thus, \begin{align} \frac{d \ln L(x_1, x_2, \cdots, x_n; \theta)}{d\theta}= \bigg({\sum_{i=1}^n x_i-n} \bigg) \cdot \frac{-1}{1-\theta}+ \frac{n} {\theta}. \end{align} By setting the derivative to zero, we can check that the maximizing value of $\theta$ is given by \begin{align} \hat{\theta}_{ML}= \frac{n} {\sum_{i=1}^n x_i}. \end{align} Thus, the MLE can be written as \begin{align} \hat{\Theta}_{ML}= \frac{n} {\sum_{i=1}^n X_i}. \end{align}


Problem

Let $X_1$, $X_2$, $X_3$, $...$, $X_n$ be a random sample from a $Uniform(0,\theta)$ distribution, where $\theta$ is unknown. Find the maximum likelihood estimator (MLE) of $\theta$ based on this random sample.

  • Solution
    • If $X_i \sim Uniform(0,\theta)$, then \begin{align} \nonumber f_X(x) = \left\{ \begin{array}{l l} \frac{1}{\theta} & \quad 0 \leq x \leq \theta \\ & \quad \\ 0 & \quad \text{otherwise} \end{array} \right. \end{align} The likelihood function is given by \begin{align} L(x_1, x_2, \cdots, x_n; \theta)&=f_{X_1 X_2 \cdots X_n}(x_1, x_2, \cdots, x_n; \theta)\\ &=f_{X_1}(x_1;\theta) f_{X_2}(x_2;\theta) \cdots f_{X_n}(x_n;\theta)\\ &=\left\{ \begin{array}{l l} \frac{1}{\theta^n} & \quad 0 \leq x_1, x_2, \cdots, x_n \leq \theta \\ & \quad \\ 0 & \quad \text{otherwise} \end{array} \right. \end{align} Note that $\frac{1}{\theta^n}$ is a decreasing function of $\theta$. Thus, to maximize it, we need to choose the smallest possible value for $\theta$. For $i=1,2,...,n$, we need to have $\theta \geq x_i$. Thus, the smallest possible value for $\theta$ is \begin{align} \hat{\theta}_{ML}= \max(x_1,x_2, \cdots, x_n). \end{align} Therefore, the MLE can be written as \begin{align} \hat{\Theta}_{ML}= \max(X_1,X_2, \cdots, X_n). \end{align} Note that this is one of those cases wherein $\hat{\theta}_{ML}$ cannot be obtained by setting the derivative of the likelihood function to zero. Here, the maximum is achieved at an endpoint of the acceptable interval.






The print version of the book is available through Amazon here.

Book Cover