Bayes Theorem

Most classifiers learn a direct mapping from features to class labels. Naive Bayes takes a different route: it models how each class generates features, then uses probability theory to invert the question — given these features, how likely is each class? The inversion is Bayes theorem, and understanding it precisely is worth the time before touching any classifier.

Anchor problem: A medical test for a rare disease. Given a positive test result, what is the actual probability of having the disease?

# Disease prevalence (prior): 1% of population
# Test sensitivity: 95% — if you have disease, test is positive 95% of the time
# Test specificity: 90% — if you don't have disease, test is negative 90% of the time
# Test false positive rate: 1 - specificity = 10%

P_disease        = 0.01   # P(D)
P_no_disease     = 0.99   # P(¬D)
P_pos_given_D    = 0.95   # P(+|D)  — sensitivity
P_pos_given_noD  = 0.10   # P(+|¬D) — false positive rate

Bayes Theorem — Statement

$P(A\mid B)=\frac{P(B\mid A)\times P(A)}{P(B)}$

In English: the probability of A given B equals the probability of B given A, times our prior belief in A, divided by the overall probability of B.

For the disease problem:

• A = "has disease" (D)
• B = "positive test result" (+)

$P(D\mid +)=\frac{P(+\mid D)\times P(D)}{P(+)}$

Each term has a name:

Step 1: The Prior

$P(D)=0.01$ — only 1 in 100 people in the population has this disease.

This number does enormous work. Before you walk into the testing clinic, there is a 99% chance you don't have the disease. A test result, no matter how accurate, must shift you away from 99% healthy — not drag you to "probably sick." Many people (and clinical practitioners) intuitively skip this step and assume a positive test implies high likelihood of disease.

Step 2: The Likelihood

$P(+\mid D)=0.95$ — the test catches 95% of sick patients. This is the test's sensitivity.

$P(+\mid \neg D)=0.10$ — 10% of healthy people also test positive. This is the false positive rate (1 − specificity).

Step 3: The Marginal (Total Probability of Testing Positive)

By the law of total probability — positive tests come from two sources: sick people who test positive, and healthy people who test positive:

$P(+)=P(+\mid D)\times P(D)+P(+\mid \neg D)\times P(\neg D)$

$=0.95\times 0.01+0.10\times 0.99=0.0095+0.0990=0.1085$

Only 10.85% of all people tested will test positive. The denominator is dominated by the second term: healthy people who falsely test positive.

Step 4: The Posterior

$P(D\mid +)=\frac{P(+\mid D)\times P(D)}{P(+)}=\frac{0.95\times 0.01}{0.1085}=\frac{0.0095}{0.1085}=0.0876$

Even with a positive test, there is only an 8.76% chance of actually having the disease.

The positive test raised the probability from 1% (prior) to 8.76% (posterior) — an 8.76× update. But 91.24% of positive tests are still false positives. This is not a flaw in the test; it is the mathematical consequence of a low base rate.

P_D  = 0.01
P_nD = 0.99
P_pos_D  = 0.95
P_pos_nD = 0.10

P_pos = P_pos_D * P_D + P_pos_nD * P_nD
P_D_given_pos = (P_pos_D * P_D) / P_pos

print(f"P(+)    = {P_pos:.4f}")
print(f"P(D|+)  = {P_D_given_pos:.4f} = {P_D_given_pos*100:.2f}%")
print(f"P(¬D|+) = {1-P_D_given_pos:.4f} = {(1-P_D_given_pos)*100:.2f}%")

Population Counting — Making This Concrete

Instead of fractions, imagine 10,000 people screened:

$P(D\mid +)=95/1085=8.76\%$ — same answer, now visually clear.

The blue box encloses all 1,085 positive tests. Only 95 of them (the blue TP box, top-left) are true positives. The red FP box contributes 990 — more than 10× the true positives.

Effect of Changing the Prior

The posterior depends heavily on the prior. A test with the same sensitivity (95%) and specificity (90%) gives dramatically different posteriors depending on disease prevalence:

priors = [0.001, 0.01, 0.05, 0.10, 0.50]
print(f"{'Prior P(D)':>12} {'P(+)':>8} {'Posterior P(D|+)':>18}")
for P_D in priors:
    P_pos = 0.95*P_D + 0.10*(1-P_D)
    posterior = (0.95 * P_D) / P_pos
    print(f"{P_D:>12.3f} {P_pos:>8.4f} {posterior:>18.4f} ({posterior*100:.1f}%)")

At prior = 0.1%, a positive test barely moves the needle (0.9% posterior). At prior = 50%, a positive test is almost definitive (90.5%). The test sensitivity and specificity are fixed — the prior does the heavy lifting.

Connecting to ML: Generative vs Discriminative

Bayes theorem is the foundation of generative classifiers: models that explicitly learn $P(x\mid y)$ (how features are distributed within each class) and $P(y)$ (class prior), then infer the class label via:

$P(y\mid x)=\frac{P(x\mid y)\times P(y)}{P(x)}$

Naive Bayes, Linear Discriminant Analysis, and Hidden Markov Models are all generative.

Discriminative classifiers (logistic regression, SVM, random forests) skip the generative model and directly learn $P(y\mid x)$ from training data. They don't need to know how features were generated — they just need to learn the decision boundary.

Generative models require stronger assumptions but generalize better with less data. Discriminative models are more flexible but need more samples to estimate the decision boundary well.

Test Your Understanding

1. The posterior $P(D\mid +)=8.76\%$. After a second independent positive test, what is the new posterior? Use the first posterior (8.76%) as the new prior for the second test — compute $P(D\mid ++$).

2. Test specificity increases from 90% to 99% (false positive rate drops from 10% to 1%). Recompute $P(D\mid +)$ for the original disease prevalence of 1%. How does this compare to 8.76%?

3. In the population of 10,000: 95 true positives and 990 false positives. If you screen only a high-risk sub-population where prevalence is 10% (instead of 1%), how many true and false positives would you expect? What is $P(D\mid +)$ in this sub-population?

4. The formula $P(y\mid x)\propto P(x\mid y)\times P(y)$ assumes independent class posterior computation. If we have 3 classes and compute the un-normalized posterior for each, how do we normalize to get probabilities summing to 1?

5. Generative classifiers model $P(x\mid y)$, which requires specifying how features are distributed. Naive Bayes assumes Gaussian or Multinomial distributions. What goes wrong if the actual feature distribution is heavily skewed and you use a Gaussian assumption?