1.4 Total probability theorem and bayes' rule

---

Total Probability Theorem

 

Let \(A_1, \cdots , \ A_n\) be disjoint events that form a partition of the sample space (each possible outcome is included in exactly one of the events A1, ... , An) and assume that \(P(A_i) > 0\), for all \(i\). Then, for any event B, we have

$$P(B) = P(A_1 \cap B) + ... + P(A_n \cap B) = P(A_1)P(B I A_1) + ... + P(A_n )P(B I A_n).$$

---

 

The theorem is visualized and proved in Fig. 1.13.
Intuitively, we are partitioning the sample space into a number of scenarios (events) \(A_i\) .

Then, the probability that B occurs is a weighted average of its conditional probability under each scenario, where each scenario is weighted according to its (unconditional) probability.

One of the uses of the theorem is to compute the probability of various events B for which the conditional probabilities \(P(B|A_i)\) are known or easy to derive. The key is to choose appropriately the \(A_1 ..... A_n\). and this choice is by problem structure. Here are some examples.

 

Figure 1.13: Visualization and verification of the total theorem.

 

The events A1, . . . . An form a partition of the sample space. so the event B can be decomposed into the disjoint union of its intersections \(A_1 \cap B\) with the sets \(A_1\).

i.e., $$B = (A_1 \cap B) \cup \cdots \cup (A_n \cap B).$$

Using the additivity axiom, it follows that $$P(B) = P(A_1 \cap B) + \cdots + P(A_n \cap B)$$.

Since, by the definition of conditional probability, we have $$P(A_i \cap B) = P(A_i)P(B|A_i).$$

the preceding equality yields $$P(B) = P(A_1)P(B|A_1)+ \cdots +P(A_n)P(B|A_n).$$

 

 

 

For an alternative view. consider an equivalent sequential model, as shown on the right.

The probability of the leaf \(A_i \cap B\) is the product \(P(A_i)P(B | A_i)\) of the probabilities along the path leading to that leaf. The event B consists of the three highlighted leaves and \(P(B)\) is obtained by adding their probabilities.

 

---

Ex 1.13: chess tournament

type 1: your probability of winning a game is 0.3 against half the players

type 2: 0.4 against a quater of the players

type 3: 0.5 against the remaining quarter of the players

 

You play a game against a randomly chosen apponent.

To find: the probability of winning?

 

Let \(A_i\) be the event of playing with an opponent of type \(i\). We have

$$P(A_1) = 0.5, \ \ \ \ \ P(A_2) = 0.25, \ \ \ \ \ P(A_3) = 0.25.$$

Also, let B be the event of winning. We have

$$P(B|A_1) = 0.3, \ \ \ \ \ P(B|A_2) = 0.4, \ \ \ \ \ P(B|A_3) = 0.5.$$

 

 

Thus, by the total probability theorem, the probability of winning is

 

$$P(B) = P(A_1)P(B|A_1)+P(A_2)P(B|A_2) + P(A_3)P(B|A_3)$$

$$=0.5 \cdot 0.3 + 0.25 \cdot 0.4 + 0.25 \cdot 0.5$$

$$= 0.375$$

---

Ex 1.14: roll a fair four-sided die

 

 

---

Ex 1.15

 

 

 

---

Inference and Bayes' Rule

The total probability theorem is often used in conjunction with the following celebrated theorem, which relates conditional probabilities of the form \(P(A|B)\) with conditional probabilities of the form \(P(B|A)\), in which the order of the conditioning is reversed.

 

---

Bayes' Rule

 

Let \(A_1, A_2, ... , A_n\) be disjoint events that form a partition of the sample space, and assume that \(P(A_i) > 0\), for all \(i\). Then, for any event B such that \(P(B) > 0\), we have

$$P(A_i|B) = \frac{P(A_i)P(B|A_i)}{P(B)} = \frac{P(A_i)P(B|A_i)}{P(A_1)P(B|A_1)+ \cdots + P(A_n)P(B|A_n).}$$

---

To verify Bayes' rule,

• 1st equality : note that by the definition of conditional probability, we have $$P(A_i \cap B) = P(A_i)P(B|A_i) = P(A_i|B)P(B).$$
• 2nd equality: follows from the first by using the total probability theorem to rewrite \(P(B)\).

 

 

Used for inference

There are a number of "causes" that may result in a certain "effect."

We observe the effect, and we wish to infer the cause.

The events A1, ... , An are associated with the causes and the event B represents the effect.

The probability \(P(B|A_i)\) that the effect will be observed when the cause \(A_i\) is present amounts to a probabilistic model of the cause-effect relation (cf. Fig. 1.14).

 

 

Given that the effect B has been observed,
we wish to evaluate the probability \(P(A_i | B)\) that the cause \(A_i\) is present.

We refer to \(P(A_i|B)\) as the posterior probability of event \(A_i\) given the information,
to be distinguished from \(P(A_i)\), which we call the prior probability.

 

 

---

Figure 1.14: An example of the inference context that is implicit in Bayes' rule.

We observe a shade in a person's (this is event B, the "effect" ) and
we want to estimate the likelihood of three mutually exclusive and collectively exhaustive potential causes:

• cause 1 ( event A1): there is a malignant  tumor,
• cause 2 (event A2): there is a non-malignant tumor,
• cause 3 ( event A3): corresponds to reasons other than a tumor.

We assume that we know the probabilities \(P(A_i)\) and \(P(B|A_i), i = 1, 2, 3\).

Bayes' rule gives the posterior probabilities of the various causes as 

$$P(A_i|B) = \frac{P(A_i)P(B|A_i)}{P(A_1)P(B|A_1) + P(A_2)P(B|A_2) + P(A_3)P(B|A_3)}, i = 1, 2, 3.$$

 

The probability \(P(A_1|B)\) of a malignant tumor is the probability of the first highlighted leaf, which is \(P(A_1 \cap B)\), divided by the total probability of the highlighted leaves, which is \(P(B)\).

 

 

 

---

Ex 1.16: Radar detection

Let us return to the radar detection problem of Example 1.9 and Fig. 1 .9. Let

$$A = \text{{an aircraft is present},}$$

$$B = \text{{the radar generates an alarm}.}$$

 

We are given that

 

$$P(A) = 0.05, \ \ \ \ \ \ P(B|A) = 0.99, \ \ \ \ \ \ P(B|A^c) =0.1.$$

Applying Bayes' rule, with \(A_1 = A\) and \(A_2 = A^c\), we obtain

\begin{align*}
 P( \text{aircraft present} | \text{alarm})&= P(A|B)\\
 &= \frac{P(A)P(B|A)}{P(B)}\\
 &= \frac{P(A)P(B|A)}{P(A)P(B|A)+P(A^c)P(B|A^c)}\\
 &= \frac{0.05 \cdot 0.99}{0.05 \cdot 0.99+095 \cdot 0.1}\\
 &\approx 0.3426 
\end{align*} 

 

---

Ex 1.17: chess tournament

Let us return to the chess problem of Example 1 .13. Here. At is the event of getting an opponent of type i, and

$$P(A_1) = 0.5, \ \ \ \  P(A_2) = 0.25, \ \ \ \  P(A_3) = 0.25.$$

 

---

Ex 1.18: The False-Positive Puzzle

• Event A: the person has the disease
• Event B: the test results are positive

A test for a certain rare disease is assumed to be correct 95% of the time:

if a person has the disease, the test results are positive with probability 0.95

if the person does not have the disease, the test results are negative with probability 0.95.

 

A random person drawn from a certain population has probability 0.001 of having the disease.

 

To find: Given that the person just tested positive, what is the probability of having the disease?

 

 

$$\begin{align*}
 P( A | B)&= \frac{P(A)P(B|A)}{P(A)P(B|A)+P(A^c)P(B|A^c)}\\
 &= \frac{0.001 \cdot 0.95}{0.001 \cdot 0.95+0999 \cdot 0.05}\\
 &= 0.0187
\end{align*} $$

 

Note that even though the test was assumed to be fairly accurate, a person who has tested positive is still very unlikely (less than 2%) to have the disease.

---

References

Bertsekas, D. P., Tsitsiklis, J. N. (2008). Introduction to Probability Second Edition. Athena Scientific.

Lecture 2: Conditioning and Bayes' Rule | Probabilistic Systems Analysis and Applied Probability | Electrical Engineering and Computer Science | MIT OpenCourseWare