8.3.3 Confidence Intervals for Normal Samples
Chi-Squared Distribution
Let us remember the gamma distribution. A continuous random variable $X$ is said to have a gamma distribution with parameters $\alpha>0$ and $\lambda>0$, shown as $X \sim Gamma(\alpha, \lambda)$, if its PDF is given by \begin{equation} \nonumber f_X(x) = \left\{ \begin{array}{l l} \frac{\lambda^{\alpha} x^{\alpha-1} e^{-\lambda x}}{\Gamma(\alpha)}& \quad x>0\\ 0 & \quad \textrm{otherwise} \end{array} \right. \end{equation} Now, we would like to define a closely related distribution, called the chi-squared distribution. We know that if $Z_1$, $Z_2$, $\cdots$, $Z_n$ are independent standard normal random variables, then the random variable \begin{align}%\label{} X=Z_1+Z_2+\cdots+Z_n \end{align} is also normal. More specifically, $X \sim N(0, n)$. Now, if we define a random variable $Y$ as \begin{align}%\label{} Y=Z^2_1+Z^2_2+\cdots+Z^2_n, \end{align} then $Y$ is said to have a chi-squared distribution with $n$ degrees of freedom shown by \begin{align}%\label{} Y \sim \chi^2(n). \end{align} It can be shown that the random variable $Y$ has, in fact, a gamma distribution with parameters $\alpha=\frac{n}{2}$ and $\lambda=\frac{1}{2}$, \begin{align}%\label{} Y \sim Gamma\left(\frac{n}{2},\frac{1}{2}\right). \end{align} Figure 8.5 shows the PDF of $\chi^2(n)$ distribution for some values of $n$.
The Chi-Squared Distribution
Definition . If $Z_1$, $Z_2$, $\cdots$, $Z_n$ are independent standard normal random variables, the random variable $Y$ defined as \begin{align}%\label{} Y=Z^2_1+Z^2_2+\cdots+Z^2_n \end{align} is said to have a chi-squared distribution with $n$ degrees of freedom shown by \begin{align}%\label{} Y \sim \chi^2(n). \end{align} Properties:- The chi-squared distribution is a special case of the gamma distribution. More specifically, \begin{align}%\label{} Y \sim Gamma\left(\frac{n}{2},\frac{1}{2}\right). \end{align} Thus, \begin{equation} \nonumber f_Y(y) = \frac{1}{2^{\frac{n}{2}} \Gamma\left(\frac{n}{2}\right)} y^{\frac{n}{2}-1} e^{-\frac{y}{2}}, \quad \textrm{for } y>0. \end{equation}
- $EY=n$, $\textrm{Var}(Y)=2n$.
- For any $p \in [0,1]$ and $n \in \mathbb{N}$, we define $\chi^2_{p,n}$ as the real value for which \begin{align}%\label{} P(Y > \chi^2_{p,n})=p, \end{align} where $Y \sim \chi^2(n)$. Figure 8.6 shows $\chi^2_{p,n}$. In MATLAB, to compute $\chi^2_{p,n}$ you can use the following command: $\mathtt{chi2inv(1-p,n)}$
Theorem .
Let $X_1$, $X_2$, $\cdots$, $X_n$ be i.i.d. $N(\mu, \sigma^2)$ random variables. Also, let $S^2$ be the sample variance for this random sample. Then, the random variable $Y$ defined as
\begin{equation}
Y=\frac{(n-1)S^2}{\sigma^2}=\frac{1}{\sigma^2}\sum_{i=1}^{n} (X_i-\overline{X})^2
\end{equation}
has a chi-squared distribution with $n-1$ degrees of freedom, i.e., $Y \sim \chi^2(n-1)$. Moreover, $\overline{X}$ and $S^2$ are independent random variables.
The $t$-Distribution
The next distribution that we need is the Student's t-distribution (or simply the t-distribution). Here, we provide the definition and some properties of the $t$-distribution. $\\$The t-Distribution
Definition . Let $Z \sim N(0,1)$, and $Y \sim \chi^2(n)$, where $n \in \mathbb{N}$. Also assume that $Z$ and $Y$ are independent. The random variable $T$ defined as \begin{align}%\label{} T=\frac{Z}{\sqrt{Y / n}} \end{align} is said to have a $t$-distribution with $n$ degrees of freedom shown by \begin{align}%\label{} T \sim T(n). \end{align} Properties:- The $t$-distribution has a bell-shaped PDF centered at $0$, but its PDF is more spread out than the normal PDF (Figure 8.7).
- $ET=0$, for $n>0$. But $ET$, is undefined for $n=1$.
- $\textrm{Var}(T)=\frac{n}{n-2}$, for $n>2$. But, $\textrm{Var}(T)$ is undefined for $n=1,2$.
- As $n$ becomes large, the $t$ density approaches the standard normal PDF. More formally, we can write \begin{align}%\label{eq:union-bound} T(n) \ \xrightarrow{d}\ N(0,1). \end{align}
- For any $p \in [0,1]$ and $n \in \mathbb{N}$, we define $t_{p,n}$ as the real value for which \begin{align}%\label{} P(T > t_{p,n})=p. \end{align} Since the $t$-distribution has a symmetric PDF, we have \begin{align}%\label{} t_{1-p,n}=-t_{p,n}. \end{align} In MATLAB, to compute $t_{p,n}$ you can use the following command: $\mathtt{tinv(1-p,n)}$.
Theorem .
Let $X_1$, $X_2$, $\cdots$, $X_n$ be i.i.d. $N(\mu, \sigma^2)$ random variables. Also, let $S^2$ be the sample variance for this random sample. Then, the random variable $T$ defined as
\begin{equation}
T=\frac{\overline{X}-\mu}{S / \sqrt{n}}
\end{equation}
has a $t$-distribution with $n-1$ degrees of freedom, i.e., $T \sim T(n-1)$.
Proof: Define the random variable $Z$ as \begin{equation} Z=\frac{\overline{X}-\mu}{\sigma / \sqrt{n}}. \end{equation} Then, $Z \sim N(0,1)$. Also, define the random variable $Y$ as \begin{equation} Y=\frac{(n-1)S^2}{\sigma^2}. \end{equation} Then by Theorem Theorem 8.3, $Y \sim \chi^2(n-1)$. We conclude that the random variable \begin{align}%\label{} T=\frac{Z}{\sqrt{\frac{Y}{n-1}}}=\frac{\overline{X}-\mu}{S / \sqrt{n}} \end{align} has a $t$-distribution with $n-1$ degrees of freedom.
Confidence Intervals for the Mean of Normal Random Variables
Here, we assume that $X_1$, $X_2$, $X_3$, $...$, $X_n$ is a random sample from a normal distribution $N(\mu, \sigma^2)$, and our goal is to find an interval estimator for $\mu$. We no longer require $n$ to be large. Thus, $n$ could be any positive integer. There are two possible scenarios depending on whether $\sigma^2$ is known or not. If the value of $\sigma^2$ is known, we can easily find a confidence interval for $\mu$. This can be done using exactly the same method that we used to estimate $\mu$ for a general distribution for the case of large $n$. More specifically, we know that the random variable \begin{align}%\label{} Q=\frac{\overline{X}-\mu}{\sigma/\sqrt{n}} \end{align} has $N(0,1)$ distribution. In particular, $Q$ is a function of the $X_i$'s and $\mu$, and its distribution does not depend on $\mu$. Thus, $Q$ is a pivotal quantity, and we conclude that $\left[\overline{X}- z_{\frac{\alpha}{2}} \frac{\sigma}{\sqrt{n}} , \overline{X}+ z_{\frac{\alpha}{2}} \frac{\sigma}{\sqrt{n}}\right]$ is $(1-\alpha)100\%$ confidence interval for $\mu$.
Assumptions: A random sample $X_1$, $X_2$, $X_3$, $...$, $X_n$ is given from a $N(\mu, \sigma^2)$ distribution, where $\textrm{Var}(X_i)=\sigma^2$ is known. $\\$
Parameter to be Estimated: $\mu=EX_i$. $\\$
Confidence Interval: $\left[\overline{X}- z_{\frac{\alpha}{2}} \frac{\sigma}{\sqrt{n}} , \overline{X}+ z_{\frac{\alpha}{2}} \frac{\sigma}{\sqrt{n}}\right]$ is a $(1-\alpha)100\%$ confidence interval for $\mu$.
Parameter to be Estimated: $\mu=EX_i$. $\\$
Confidence Interval: $\left[\overline{X}- z_{\frac{\alpha}{2}} \frac{\sigma}{\sqrt{n}} , \overline{X}+ z_{\frac{\alpha}{2}} \frac{\sigma}{\sqrt{n}}\right]$ is a $(1-\alpha)100\%$ confidence interval for $\mu$.
The more interesting case is when we do not know the variance $\sigma^2$. More specifically, we are given $X_1$, $X_2$, $X_3$, $...$, $X_n$, which is a random sample from a normal distribution $N(\mu, \sigma^2)$, and our goal is to find an interval estimator for $\mu$. However, $\sigma^2$ is also unknown. In this case, using Theorem 8.4, we conclude that the random variable $T$ defined as \begin{equation} T=\frac{\overline{X}-\mu}{S / \sqrt{n}} \end{equation} has a $t$-distribution with $n-1$ degrees of freedom, i.e., $T \sim T(n-1)$. Here, the random variable $T$ is a pivotal quantity, since it is a function of the $X_i$'s and $\mu$, and its distribution does not depend on $\mu$ or any other unknown parameters. Now that we have a pivot, the next step is to find a $(1-\alpha)$ interval for $T$. Using the definition of $t_{p,n}$, a $(1-\alpha)$ interval for $T$ can be stated as \begin{align}%\label{} P\left(-t_{\frac{\alpha}{2},n-1} \leq T \leq t_{\frac{\alpha}{2},n-1} \right)= 1-\alpha. \end{align} Therefore, \begin{align}%\label{} P\left(-t_{\frac{\alpha}{2},n-1} \leq \frac{\overline{X}-\mu}{S / \sqrt{n}} \leq t_{\frac{\alpha}{2},n-1} \right)= 1-\alpha, \end{align} which is equivalent to \begin{align}%\label{} P\left(\overline{X}- t_{\frac{\alpha}{2},n-1}\frac{S}{\sqrt{n}} \leq \mu \leq \overline{X}+ t_{\frac{\alpha}{2},n-1} \frac{S}{\sqrt{n}} \right)= 1-\alpha. \end{align} We conclude that $\left[\overline{X}- t_{\frac{\alpha}{2},n-1}\frac{S}{\sqrt{n}} , \overline{X}+ t_{\frac{\alpha}{2},n-1} \frac{S}{\sqrt{n}}\right]$ is $(1-\alpha)100\%$ confidence interval for $\mu$.
Assumptions: A random sample $X_1$, $X_2$, $X_3$, $...$, $X_n$ is given from a $N(\mu, \sigma^2)$ distribution, where $\mu=EX_i$ and $\textrm{Var}(X_i)=\sigma^2$ are unknown. $\\$
Parameter to be Estimated: $\mu=EX_i$. $\\$
Confidence Interval: $\left[\overline{X}- t_{\frac{\alpha}{2},n-1}\frac{S}{\sqrt{n}} , \overline{X}+ t_{\frac{\alpha}{2},n-1} \frac{S}{\sqrt{n}}\right]$ is a $(1-\alpha)$ confidence interval for $\mu$.
Parameter to be Estimated: $\mu=EX_i$. $\\$
Confidence Interval: $\left[\overline{X}- t_{\frac{\alpha}{2},n-1}\frac{S}{\sqrt{n}} , \overline{X}+ t_{\frac{\alpha}{2},n-1} \frac{S}{\sqrt{n}}\right]$ is a $(1-\alpha)$ confidence interval for $\mu$.
Example
A farmer weighs $10$ randomly chosen watermelons from his farm and he obtains the following values (in lbs): \begin{equation} 7.72 \quad 9.58 \quad 12.38 \quad 7.77 \quad 11.27 \quad 8.80 \quad 11.10 \quad 7.80 \quad 10.17 \quad 6.00 \end{equation} Assuming that the weight is normally distributed with mean $\mu$ and variance $\sigma^2$, find a $95 \%$ confidence interval for $\mu$.
- Solution
- Using the data we obtain \begin{align}%\label{} &\overline{X}=9.26,\\ &S^2=3.96 \end{align} Here, $n=10$, $\alpha=0.05$, so we need \begin{align}%\label{} t_{0.025,9} \approx 2.262 \end{align} The above value can be obtained in MATLAB using the command $\mathtt{tinv(0.975,9)}$. Thus, we can obtain a $95 \%$ confidence interval for $\mu$ as \begin{align}%\label{} \left[\overline{X}- t_{\frac{\alpha}{2},n-1}\frac{S}{\sqrt{n}} , \overline{X}+ t_{\frac{\alpha}{2},n-1} \frac{S}{\sqrt{n}}\right] &=\\ \hspace{-60pt}&=\left [9.26-2.26 \cdot \frac{\sqrt{3.96}}{\sqrt{10}}, 9.26+2.26 \cdot \frac{\sqrt{3.96}}{\sqrt{10}}\right]\\ \hspace{-60pt}&=[7.84, 10.68]. \end{align} Therefore, $[7.84, 10.68]$ is a $95 \%$ confidence interval for $\mu$.
Confidence Intervals for the Variance of Normal Random Variables
Now, suppose that we would like to estimate the variance of a normal distribution. More specifically, assume that $X_1$, $X_2$, $X_3$, $...$, $X_n$ is a random sample from a normal distribution $N(\mu, \sigma^2)$, and our goal is to find an interval estimator for $\sigma^2$. We assume that $\mu$ is also unknown. Again, $n$ could be any positive integer. By Theorem 8.3, the random variable $Q$ defined as \begin{equation} Q=\frac{(n-1)S^2}{\sigma^2}=\frac{1}{\sigma^2}\sum_{i=1}^{n} (X_i-\overline{X})^2 \end{equation} has a chi-squared distribution with $n-1$ degrees of freedom, i.e., $Q \sim \chi^2(n-1)$. In particular, $Q$ is a pivotal quantity since it is a function of the $X_i$'s and $\sigma^2$, and its distribution does not depend on $\sigma^2$ or any other unknown parameters. Using the definition of $\chi^2_{p,n}$, a $(1-\alpha)$ interval for $Q$ can be stated as \begin{align}%\label{} P\left(\chi^2_{1-\frac{\alpha}{2},n-1} \leq Q \leq \chi^2_{\frac{\alpha}{2},n-1} \right)= 1-\alpha. \end{align} Therefore, \begin{align}%\label{} P\left(\chi^2_{1-\frac{\alpha}{2},n-1} \leq \frac{(n-1)S^2}{\sigma^2} \leq \chi^2_{\frac{\alpha}{2},n-1} \right)= 1-\alpha. \end{align} which is equivalent to \begin{align}%\label{} P\left(\frac{(n-1)S^2}{\chi^2_{\frac{\alpha}{2},n-1}} \leq \sigma^2 \leq \frac{(n-1)S^2}{\chi^2_{1-\frac{\alpha}{2},n-1}} \right)= 1-\alpha. \end{align} We conclude that $\left[\frac{(n-1)S^2}{\chi^2_{\frac{\alpha}{2},n-1}} , \frac{(n-1)S^2}{\chi^2_{1-\frac{\alpha}{2},n-1}} \right]$ is a $(1-\alpha)100\%$ confidence interval for $\sigma^2$.
Assumptions: A random sample $X_1$, $X_2$, $X_3$, $...$, $X_n$ is given from a $N(\mu, \sigma^2)$ distribution, where $\mu=EX_i$ and $\textrm{Var}(X_i)=\sigma^2$ are unknown. $\\$
Parameter to be Estimated: $\textrm{Var}(X_i)=\sigma^2$. $\\$
Confidence Interval: $\left[\frac{(n-1)S^2}{\chi^2_{\frac{\alpha}{2},n-1}} , \frac{(n-1)S^2}{\chi^2_{1-\frac{\alpha}{2},n-1}} \right]$ is a $(1-\alpha)100\%$ confidence interval for $\sigma^2$.
Parameter to be Estimated: $\textrm{Var}(X_i)=\sigma^2$. $\\$
Confidence Interval: $\left[\frac{(n-1)S^2}{\chi^2_{\frac{\alpha}{2},n-1}} , \frac{(n-1)S^2}{\chi^2_{1-\frac{\alpha}{2},n-1}} \right]$ is a $(1-\alpha)100\%$ confidence interval for $\sigma^2$.
Example
For the data given in Example 8.20, find a $95 \%$ confidence interval for $\sigma^2$. Again, assume that the weight is normally distributed with mean $\mu$ and variance $\sigma^2$, where $\mu$ and $\sigma$ are unknown.
- Solution
- As before, using the data we obtain \begin{align}%\label{} &\overline{X}=9.26,\\ &S^2=3.96 \end{align} Here, $n=10$, $\alpha=0.05$, so we need \begin{align}%\label{} \chi^2_{0.025,9}=19.02, \quad \chi^2_{0.975,9}= 2.70 \end{align} The above values can obtained in MATLAB using the commands $\mathtt{chi2inv(0.975,9)}$ and $\mathtt{chi2inv(0.025,9)}$, respectively. Thus, we can obtain a $95 \%$ confidence interval for $\sigma^2$ as \begin{align}%\label{} \left[\frac{(n-1)S^2}{\chi^2_{\frac{\alpha}{2},n-1}} , \frac{(n-1)S^2}{\chi^2_{1-\frac{\alpha}{2},n-1}} \right] &=\left[\frac{9 \times 3.96}{19.02} , \frac{9 \times 3.96}{2.70} \right]\\ &=[1.87 , 13.20]. \end{align} Therefore, $[1.87 , 13.20]$ is a $95 \%$ confidence interval for $\sigma^2$.
