---

Here we would like to discuss how we find interval estimators. Before doing so, let's review a simple fact from random variables and their distributions. Let $X$ be a continuous random variable with CDF $F_X(x)=P(X \leq x)$. Suppose that we are interested in finding two values $x_h$ and $x_l$ such that \begin{align}%\label{} P\bigg(x_l \leq X \leq x_h \bigg)= 1-\alpha. \end{align} One way to do this, is to chose $x_l$ and $x_h$ such that \begin{align}%\label{} P\big(X \leq x_l \big)= \frac{\alpha}{2}, \quad \textrm{and} \quad P\big(X \geq x_h)= \frac{\alpha}{2}. \end{align} Equivalently, \begin{align}%\label{} F_X(x_l)=\frac{\alpha}{2}, \quad \textrm{and} \quad F_X(x_h)= 1-\frac{\alpha}{2}. \end{align} We can rewrite these equations by using the inverse function $F^{-1}_X$ as \begin{align}%\label{} x_l= F^{-1}_X\left(\frac{\alpha}{2}\right), \quad \textrm{and} \quad x_h= F^{-1}_X\left(1-\frac{\alpha}{2}\right). \end{align} We call the interval $[x_l, x_h]$ a ($1-\alpha$) interval for $X$. Figure 8.2 shows the values of $x_l$ and $x_h$ using the CDF of $X$, and also using the PDF of $X$.

Example
Let $Z \sim N(0,1)$, find $x_l$ and $x_h$ such that \begin{align}%\label{} P\bigg(x_l \leq Z \leq x_h \bigg)= 0.95 \end{align}

---

Now, let's talk about how we can find interval estimators. A general approach is to start with a point estimator $\hat{\Theta}$, such as the MLE, and create the interval $\big[\hat{\Theta}_l,\hat{\Theta}_h\big]$ around it such that $P\bigg(\theta \in \big[\hat{\Theta}_l,\hat{\Theta}_h\big] \bigg) \geq 1-\alpha$. How do we do this? Let's look at an example.

---

Example
Let $X_1$, $X_2$, $X_3$, $...$, $X_n$ be a random sample from a normal distribution $N(\theta, 1)$. Find a $95 \%$ confidence interval for $\theta$.

---

At first, it might seem that our solution to Example 8.13 is not based on a systematic method. You might have asked: "How should I know that I need to work with the normalized $\overline{X}$?" However, by thinking more deeply about the way we solved this example, we can suggest a general method to solve confidence interval problems. The crucial fact about the random variable \begin{align}%\label{} \overline{X}-\theta \end{align} is that its distribution does not depend on the unknown parameter $\theta$. Thus, we could easily find a $95\%$ interval for the random variable $\sqrt{n}(\overline{X}-\theta)$ that did not depend on $\theta$. Such a random variable is called a pivot or a pivotal quantity. Let us define this more precisely. Example
Check that the random variables $Q_1=\overline{X}-\theta$ and $Q_2=\sqrt{n}(\overline{X}-\theta)$ are both valid pivots in Example 8.13.

---

To summarize, here are the steps in the pivotal method for finding confidence intervals:

1. First, find a pivotal quantity $Q(X_1,X_2, \cdots, X_n, \theta)$.
2. Find an interval for $Q$ such that \begin{align}%\label{} P\big(q_l \leq Q \leq q_h\big)= 1-\alpha. \end{align}
3. Using algebraic manipulations, convert the above equation to an equation of the form \begin{align}%\label{} P\big(\hat{\Theta}_l \leq \theta \leq \hat{\Theta}_h\big)= 1-\alpha. \end{align}

You are probably still not sure how exactly you can perform these steps. The most crucial one is the first step. How do we find a pivotal quantity? Luckily, for many important cases that appear frequently in practice, statisticians have already found the pivotal quantities, so we can use their results directly. In practice, many of the interval estimation problems you encounter are of the forms for which general confidence intervals have been found previously. Therefore, to solve many confidence interval problems, it suffices to write the problem in a format similar to a previously solved problem. As you see more examples, you will feel more confident about solving confidence interval problems.

---

Example
Let $X_1$, $X_2$, $X_3$, $...$, $X_n$ be a random sample from a distribution with known variance $\textrm{Var}(X_i)=\sigma^2$, and unknown mean $EX_i=\theta$. Find a $(1-\alpha)$ confidence interval for $\theta$. Assume that $n$ is large.

---

The above example is our first important case of known interval estimators, so let's summarize what we have shown: Note that to obtain the above interval, we used the CLT. Thus, what we found is an approximate confidence interval. Nevertheless, for large $n$, the approximation is very good.

---

Example
An engineer is measuring a quantity $\theta$. It is assumed that there is a random error in each measurement, so the engineer will take $n$ measurements and report the average of the measurements as the estimated value of $\theta$. Here, $n$ is assumed to be large enough so that the central limit theorem applies. If $X_i$ is the value that is obtained in the $i$th measurement, we assume that \begin{align}%\label{} X_i=\theta+W_i, \end{align} where $W_i$ is the error in the $i$th measurement. We assume that the $W_i$'s are i.i.d. with $EW_i=0$ and $\textrm{Var}(W_i)=4$ square units. The engineer reports the average of the measurements \begin{align}%\label{} \overline{X}=\frac{X_1+X_2+...+X_n}{n}. \end{align} How many measurements does the engineer need to make until he is $90\%$ sure that the final error is less than $0.25$ units? In other words, what should the value of $n$ be such that \begin{equation} P\big(\theta-0.25 \leq \overline{X} \leq \theta+0.25\big) \geq .90 \, ? \end{equation}

---

Now suppose that $X_1$, $X_2$, $X_3$, $...$, $X_n$ is a random sample from a distribution with unknown variance $\textrm{Var}(X_i)=\sigma^2$. Our goal is to find a $1-\alpha$ confidence interval for $\theta=EX_i$. We also assume that $n$ is large. By the above discussion, we can say \begin{align}%\label{} P\left(\overline{X}- z_{\frac{\alpha}{2}} \frac{\sigma}{\sqrt{n}} \leq \theta \leq \overline{X}+ z_{\frac{\alpha}{2}} \frac{\sigma}{\sqrt{n}} \right)= 1-\alpha. \end{align} However, there is a problem here. We do not know the value of $\sigma$. How do we deal with this issue? There are two general approaches: we can either find an upper bound for $\sigma$, or we can estimate $\sigma$.

1. An upper bound for $\sigma^2$: Suppose that we can somehow show that \begin{align}%\label{} \sigma \leq \sigma_{max}, \end{align} where $\sigma_{max}<\infty$ is a real number. Then, if we replace $\sigma$ in $\left[\overline{X}- z_{\frac{\alpha}{2}} \frac{\sigma}{\sqrt{n}} , \overline{X}+ z_{\frac{\alpha}{2}} \frac{\sigma}{\sqrt{n}}\right]$ by $\sigma_{max}$, the interval gets bigger. In other words, the interval \begin{align}%\label{} \left[\overline{X}- z_{\frac{\alpha}{2}} \frac{\sigma_{max}}{\sqrt{n}} , \overline{X}+ z_{\frac{\alpha}{2}} \frac{\sigma_{max}}{\sqrt{n}}\right] \end{align} is still a valid $(1-\alpha)100\%$ confidence interval for $\theta$.
2. Estimate $\sigma^2$: Note that here, since $n$ is large, we should be able to find a relatively good estimate for $\sigma^2$. After estimating $\sigma^2$, we can use that estimate and $\left[\overline{X}- z_{\frac{\alpha}{2}} \frac{\sigma}{\sqrt{n}} , \overline{X}+ z_{\frac{\alpha}{2}} \frac{\sigma}{\sqrt{n}}\right]$ to find an approximate $(1-\alpha)100\%$ confidence interval for $\theta$.

We now provide examples of each approach.

---

Example
(Public Opinion Polling) We would like to estimate the portion of people who plan to vote for Candidate A in an upcoming election. It is assumed that the number of voters is large, and $\theta$ is the portion of voters who plan to vote for Candidate A. We define the random variable $X$ as follows. A voter is chosen uniformly at random among all voters and we ask her/him: "Do you plan to vote for Candidate $A$?" If she/he says "yes," then $X=1$, otherwise $X=0$. Then, \begin{align}%\label{} X \sim Bernoulli(\theta). \end{align} Let $X_1$, $X_2$, $X_3$, $...$, $X_n$ be a random sample from this distribution, which means that the $X_i$'s are i.i.d. and $X_i \sim Bernoulli(\theta)$. In other words, we randomly select $n$ voters (with replacement) and we ask each of them if they plan to vote for Candidate A. Find a $(1-\alpha)100\%$ confidence interval for $\theta$ based on $X_1$, $X_2$, $X_3$, $...$, $X_n$.

---

The above setting is another important case of known interval estimators, so let's summarize it:

---

Example
There are two candidates in a presidential election: Candidate A and Candidate B. Let $\theta$ be the portion of people who plan to vote for Candidate A. Our goal is to find a confidence interval for $\theta$. Specifically, we choose a random sample (with replacement) of $n$ voters and ask them if they plan to vote for Candidate A. Our goal is to estimate the $\theta$ such that the margin of error is $3$ percentage points. Assume a $95 \%$ confidence level. That is, we would like to choose $n$ such that \begin{align}%\label{} P\left(\overline{X}-0.03 \leq \theta \leq \overline{X}+ 0.03 \right) \geq 0.95, \end{align} where $\overline{X}$ is the portion of people in our random sample that say they plan to vote for Candidate A. How large does $n$ need to be?

---

As we mentioned, the above calculation might be a little conservative. Another approach would be to estimate $\sigma^2$ instead of using an upper bound. In this example, the structure of the problem suggests a way to estimate $\sigma^2$. Specifically, since \begin{align}%\label{} \sigma^2=\theta(1-\theta), \end{align} we may use \begin{align}%\label{} \hat{\sigma}^2&= \hat{\theta}(1-\hat{\theta})\\ &=\overline{X}(1-\overline{X}) \end{align} as an estimate for $\theta$, where $\hat{\theta}=\overline{X}$. The rationale behind this approximation is that since $n$ is large, $\overline{X}$ is likely a good estimate of $\theta$, thus $\hat{\sigma}^2= \hat{\theta}(1-\hat{\theta})$ is a good estimate of $\sigma^2$. After estimating $\sigma^2$, we can use $\left[\overline{X}- z_{\frac{\alpha}{2}} \frac{\hat{\sigma}}{\sqrt{n}} , \overline{X}+ z_{\frac{\alpha}{2}} \frac{\hat{\sigma}}{\sqrt{n}}\right]$ as an approximate $(1-\alpha)100\%$ confidence interval for $\theta$. To summarize, we have the following confidence interval rule: Again, the above confidence interval is an approximate confidence interval because we used two approximations: the CLT and an approximation for $\sigma^2$.

---

The above scenario is a special case ($Bernoulli(\theta)$) for which we could come up with a point estimator for $\sigma^2$. Can we have a more general estimator for $\sigma^2$ that we can use for any distribution? We have already discussed such a point estimator and we called it the sample variance: \begin{align}%\label{} {S}^2=\frac{1}{n-1} \sum_{k=1}^n (X_k-\overline{X})^2=\frac{1}{n-1} \left(\sum_{k=1}^n X^2_k-n\overline{X}^2\right). \end{align} Thus, using the sample variance, ${S}^2$, we can have an estimate for $\sigma^2$. If $n$ is large, this estimate is likely to be close to the real value of $\sigma^2$. So let us summarize this discussion as follows:

---

Example
We have collected a random sample $X_1$, $X_2$, $X_3$, $...$, $X_{100}$ from an unknown distribution. The sample mean and the sample variance for this random sample are given by \begin{equation} \overline{X}=15.6, S^2=8.4 \end{equation} Construct an approximate $99 \%$ confidence interval for $\theta=EX_i$.

---