The law of total probability and Bayes' theorem – Elementary and Conditional Probability (Senior 5 Mathematics) | Ganzaa

Elementary and Conditional Probability

Lessons

Senior 5 · Mathematics

Elementary and Conditional Probability

4 lessons

Lesson 4 of 4

The law of total probability and Bayes' theorem

45 min

A positive test: how worried should you be?

A screening test for a disease is 95% accurate on sick people and 90% accurate on healthy ones (illustrative figures). A person tests positive. Most people guess that the chance of disease is about 90%. The true answer, for a disease that affects 2% of people, is about 16%. The tool that gets it right reverses a conditional probability: from $P(\text{positive} \mid \text{disease})$ , which the test maker knows, to $P(\text{disease} \mid \text{positive})$ , which the patient wants. It is named after Thomas Bayes, whose essay on it was published in 1763, after his death.

Definition

Events $B_1, B_2, \dots, B_n$ form a partition of $\Omega$ if they are mutually exclusive, cover $\Omega$ ( $B_1 \cup \dots \cup B_n = \Omega$ ), and each has $P (B_{i}) > 0$ . Exactly one of them occurs. The simplest partition is $B$ and $B^{'}$ .

Important

Law of total probability. If $B_1, \dots, B_n$ is a partition, then for any event $A$

P(A) = P(B_1)P(A \mid B_1) + P(B_2)P(A \mid B_2) + \dots + P(B_n)P(A \mid B_n) .

Reason. $A$ is the union of the non-overlapping pieces $A \cap B_i$ , and $P(A \cap B_i) = P(B_i)P(A \mid B_i)$ . In a tree diagram, this adds the paths that end in $A$ .

Bayes' theorem. For each $k$ ,

P(B_k \mid A) = \frac{P(B_k)\,P(A \mid B_k)}{\sum_{i=1}^{n} P(B_i)\,P(A \mid B_i)} .

Reason. $P(B_k \mid A) = \frac{P(A \cap B_k)}{P(A)}$ ; write the numerator with the multiplication rule and the denominator with the law of total probability. $P (B_{k})$ is called the prior probability and $P(B_k \mid A)$ the posterior probability, after the evidence $A$ .

Worked example 1: seed from three suppliers

A cooperative buys maize seed from three suppliers: 40% from $S_{1}$ , 35% from $S_{2}$ and 25% from $S_{3}$ . The proportions of seeds that fail to germinate are 5%, 4% and 10% (illustrative). (a) Find the probability that a seed chosen at random fails. (b) A seed fails. Find the probability that it came from $S_{3}$ .

Solution. Let $F$ = "fails". (a) By the law of total probability,

P (F) = 0.40 (0.05) + 0.35 (0.04) + 0.25 (0.10) = 0.020 + 0.014 + 0.025 = 0.059.

(b) By Bayes' theorem, $P(S_3 \mid F) = \frac{0.025}{0.059} \approx 0.424$ . Although $S_{3}$ supplies only a quarter of the seed, it is responsible for about 42% of the failures. (Similarly $P(S_1 \mid F) \approx 0.339$ and $P(S_2 \mid F) \approx 0.237$ ; the three add to 1.)

Counting people instead of probabilities

Bayes' theorem is easiest to trust when probabilities are turned into counts of an imagined group, "natural frequencies".

Worked example 2: the medical test

A disease affects 2% of a population. A test is positive for 95% of people with the disease and for 10% of people without it (illustrative). A person tests positive. Find the probability that the person has the disease.

Solution. Let $D$ = "has the disease" and $T$ = "tests positive". The partition is $D, D^{'}$ with $P (D) = 0.02$ , $P (D^{'}) = 0.98$ .

P(T) = 0.02(0.95) + 0.98(0.10) = 0.019 + 0.098 = 0.117, \qquad P(D \mid T) = \frac{0.019}{0.117} \approx 0.162 .

Only about 16%. The figure shows why: the healthy group is so large that its 10% of false positives (980 people) outnumbers the true positives (190).

Worked example 3: a second test updates the first

The person in worked example 2 takes a second, independent test of the same kind, and it is also positive. Find the new probability of disease.

Solution. Use the posterior from the first test, 0.162, as the new prior:

P(D \mid T_2) = \frac{0.162(0.95)}{0.162(0.95) + 0.838(0.10)} = \frac{0.1539}{0.1539 + 0.0838} \approx 0.648 .

Two positive results raise the probability to about 65%. This is why doctors confirm a positive screening test with a second test.

Worked example 4: which box?

Box 1 holds 3 red and 2 blue balls; box 2 holds 1 red and 4 blue. A fair coin decides the box, then one ball is drawn. It is red. Find the probability that it came from box 1.

Solution. $P(R) = \frac{1}{2} \times \frac{3}{5} + \frac{1}{2} \times \frac{1}{5} = 0.3 + 0.1 = 0.4$ , so $P(\text{box } 1 \mid R) = \frac{0.3}{0.4} = 0.75$ . Seeing red makes box 1 three times as likely as box 2.

Warning

Do not confuse $P(T \mid D)$ with $P(D \mid T)$ . A test that catches 95% of the sick does not make a positive result 95% reliable.
Never ignore the prior (the "base rate"). A rare condition stays fairly unlikely after one positive test.
Check that the posteriors over a partition add to 1.

Tip

Stuck? Imagine 10,000 people (or 1,000 seeds), split them along the tree, and count. The final answer is "the paths you want" divided by "all the paths that match the evidence".

3Blue1Brown: Bayes theorem, the geometry of changing beliefs (about 15 min). Watch the evidence shrink the space of possibilities, just like the natural-frequency tree in this lesson.

✦

Practice

Questions 1 to 3 are routine; 4 to 6 need more care; 7 asks for a proof.

A school's S5 pupils are in classes A (40%), B (35%) and C (25%). The pass rates in a test are 90%, 80% and 70%. Find the probability that a pupil chosen at random passed.
With the data of question 1, a pupil failed. Find the probability that the pupil is in class C.
Box 1 holds 4 white and 2 black balls; box 2 holds 1 white and 5 black. A die is rolled: on 1 or 2 a ball is drawn from box 1, otherwise from box 2. Find $P(\text{white})$ and $P(\text{box } 1 \mid \text{white})$ .
For the test of worked example 2, find the probability that a person who tests negative has the disease.
30% of a company's emails are spam. The word "free" appears in 40% of spam emails and in 5% of the others (illustrative). An email contains "free". Find the probability that it is spam.
In worked example 2, what would $P(D \mid T)$ be if the disease affected 20% of people instead of 2%? Comment.
Prove Bayes' theorem for the partition $B, B^{'}$ , starting from the definition of conditional probability.

Show answersHide answers

$0.4 (0.9) + 0.35 (0.8) + 0.25 (0.7) = 0.36 + 0.28 + 0.175 = 0.815$ .
$\frac{0.25 \times 0.3}{1 - 0.815} = \frac{0.075}{0.185} \approx 0.405$ .
$\frac{1}{3} \times \frac{4}{6} + \frac{2}{3} \times \frac{1}{6} = \frac{1}{3}$ ; $\frac{4}{18} \div \frac{6}{18} = \frac{2}{3}$ .
$\frac{0.02 \times 0.05}{0.02 \times 0.05 + 0.98 \times 0.90} = \frac{0.001}{0.883} \approx 0.0011$ : a negative result is very reassuring.
$\frac{0.3 \times 0.4}{0.3 \times 0.4 + 0.7 \times 0.05} = \frac{0.12}{0.155} \approx 0.774$ .
$\frac{0.2 \times 0.95}{0.2 \times 0.95 + 0.8 \times 0.1} = \frac{0.19}{0.27} \approx 0.70$ . The same test means much more when the condition is common: the prior matters.
$P(B \mid A) = \frac{P(A \cap B)}{P(A)}$ , with $P(A \cap B) = P(B)P(A \mid B)$ and, since $A = (A \cap B) \cup (A \cap B')$ , $P(A) = P(B)P(A \mid B) + P(B')P(A \mid B')$ .

1A positive test: how worried should you be?

2Counting people instead of probabilities

A positive test: how worried should you be?

Counting people instead of probabilities