Prior and Posterior

9.1.1 Prior and Posterior

Let $X$ be the random variable whose value we try to estimate. Let $Y$ be the observed random variable. That is, we have observed $Y=y$, and we would like to estimate $X$. Assuming both $X$ and $Y$ are discrete, we can write \begin{align} P (X=x|Y=y)&= \frac{P(X=x, Y=y)}{P(Y=y)}\\ &=\frac{P(Y=y|X=x)P(X=x)}{P(Y=y)}. \end{align} Using our notation for PMF and conditional PMF, the above equation can be rewritten as \begin{align} P_{X|Y}(x|y)=\frac{P_{Y|X}(y|x)P_{X}(x)}{P_{Y}(y)}. \end{align} The above equation, as we have seen before, is just one way of writing Bayes' rule. If either $X$ or $Y$ are continuous random variables, we can replace the corresponding PMF with PDF in the above formula. For example, if $X$ is a continuous random variable, while $Y$ is discrete we can write \begin{align} f_{X|Y}(x|y)=\frac{P_{Y|X}(y|x)f_{X}(x)}{P_{Y}(y)}. \end{align} To find the denominator ($P_{Y}(y)$ or $f_{Y}(y)$), we often use the law of total probability. Let's look at an example.

Example

Let $X \sim Uniform(0,1)$. Suppose that we know \begin{align} Y \; | \; X=x \quad \sim \quad Geometric(x). \end{align} Find the posterior density of $X$ given $Y=2$, $f_{X|Y}(x|2)$.

Solution
- Using Bayes' rule we have \begin{align} f_{X|Y}(x|2)&=\frac{P_{Y|X}(2|x)f_{X}(x)}{P_{Y}(2)}. \end{align} We know $ Y \; | \; X=x \quad \sim \quad Geometric(x)$, so \begin{align} P_{Y|X}(y|x)=x (1-x)^{y-1}, \quad \textrm{ for }y=1,2,\cdots. \end{align} Therefore, \begin{align} P_{Y|X}(2|x)=x (1-x). \end{align} To find $P_{Y}(2)$, we can use the law of total probability \begin{align} P_{Y}(2)&=\int_{-\infty}^{\infty} P_{Y|X}(2|x) f_X(x) \quad \textrm{d}x \\ &=\int_{0}^{1} x (1-x) \cdot 1 \quad \textrm{d}x \\ &=\frac{1}{6}. \end{align} Therefore, we obtain \begin{align} f_{X|Y}(x|2)&=\frac{x (1-x) \cdot 1}{\frac{1}{6}}\\ &= 6x(1-x), \quad \textrm{ for }0 \leq x \leq 1. \end{align}

For the remainder of this chapter, for simplicity, we often write the posterior PDF as \begin{align} f_{X|Y}(x|y)=\frac{f_{Y|X}(y|x)f_{X}(x)}{f_{Y}(y)}, \end{align} which implies that both $X$ and $Y$ are continuous. Nevertheless, we understand that if either $X$ or $Y$ is discrete, we need to replace the PDF by the corresponding PMF.

← previous

The print version of the book is available on Amazon.

Practical uncertainty: Useful Ideas in Decision-Making, Risk, Randomness, & AI