Bayes Theorem

You're at your annual checkup. The doctor orders a routine blood test. A few days later, your phone rings — it's the clinic. The test came back positive for a rare condition. Your heart sinks.

But the doctor says: "Don't panic. The test is good, but not perfect." What does that actually mean? How do you rationally interpret a positive result?

Most people — including many clinicians — intuitively think: the test is 95% accurate, so a positive result means a 95% chance of having the disease. That's wrong. The real answer is much lower, and it depends on something you probably haven't considered: how rare the disease is to begin with. This gap between intuition and mathematics is not a curiosity — it's a source of real-world misdiagnosis, unnecessary anxiety, and poor medical decisions.

This is exactly the problem Bayes theorem solves. It's a formal rule for updating what you believe when new evidence arrives — taking into account how likely that evidence was to begin with. Without it, you'd overreact to every positive test, every warning light, every red flag.

What It Is

Bayes theorem is a rule for combining what you already know (the base rate) with new evidence (the test result) to produce an updated belief. It treats classification as a generative process — instead of learning a direct boundary between classes, it models how data is produced within each class and then inverts the question.

This is NOT the same as what logistic regression or SVM do. Those are discriminative models — they learn a direct mapping from features to labels without ever modeling how features are distributed. Bayes theorem says: "Let's model how sick people produce test results and how healthy people produce test results, then use the base rate to figure out which explanation is more likely given what we observed." It's a fundamentally different approach to classification.

The key idea in symbols is:

$P(A\mid B)=\frac{P(B\mid A)\times P(A)}{P(B)}$

In plain English: the probability of A given B equals the probability of B given A, times our prior belief in A, divided by the overall probability of B.

For our disease problem, A = "has disease" (D) and B = "positive test" (+):

$P(D\mid +)=\frac{P(+\mid D)\times P(D)}{P(+)}$

Each term has a name:

Now let's compute each piece, one step at a time, on a concrete medical scenario.

Anchor dataset: A medical test for a rare disease. Prevalence is 1%, test sensitivity is 95%, test specificity is 90%.

The Plan — Four Steps to Bayes Theorem

We'll walk through the calculation in four phases. First we establish our starting belief about the disease (the prior). Then we define how the test behaves (the likelihood). Next we compute all the ways a positive test can happen (the marginal evidence). Finally we combine everything to get the answer (the posterior).

Phase 1: The Prior — Establish Your Starting Belief

Goal: Quantify what we believe before seeing any test result.

Only 1 in 100 people in the population has this disease. That's our starting point — the prior:

$P(D)=0.01$

This number does enormous work. Before you walk into the testing clinic, there's a 99% chance you're healthy. A test result, no matter how accurate, must shift you away from that 99% — it doesn't start from scratch.

P_D = 0.01    # prior: disease prevalence
P_nD = 0.99   # complement: probability of no disease

That tiny red sliver on the left — that's the entire disease population. Any evidence has to pull you away from the massive 99% healthy block.

✓ Phase 1 complete. Prior P(D) = 0.01: before any test, only a 1% chance of disease.

Phase 2: The Likelihood — Define How the Test Behaves

Goal: Quantify how accurate the test is for both sick and healthy people.

The test catches 95% of sick patients — that's the sensitivity. And 10% of healthy patients also test positive — that's the false positive rate (1 minus specificity):

$P(+\mid D)=0.95P(+\mid \neg D)=0.10$

The test is good at detecting the disease when it's present. But it also flags one in ten healthy people. With the disease being rare, that 10% of a massive healthy population will produce many more false positives than true positives — we'll see why next.

P_pos_D = 0.95   # sensitivity: P(+|D)
P_pos_nD = 0.10  # false positive rate: P(+|¬D)

The test parameters are fixed. The prior determines how they play out.

✓ Phase 2 complete. Likelihoods P(+|D) = 0.95, P(+|¬D) = 0.10.

Phase 3: The Marginal — Compute the Total Probability of a Positive Test

Goal: Find the overall probability of testing positive, from all possible sources.

Positive tests come from two sources: sick people who test positive, and healthy people who test positive. By the law of total probability:

$P(+)=P(+\mid D)\times P(D)+P(+\mid \neg D)\times P(\neg D)$

Substituting our numbers:

$P(+)=0.95\times 0.01+0.10\times 0.99=0.0095+0.0990=0.1085$

The blue sliver is the true positive contribution. The red mass is the false positive contribution — ten times larger. The denominator is dominated by false positives.

P_pos = P_pos_D * P_D + P_pos_nD * P_nD
print(f"P(+) = {P_pos:.4f} = {P_pos*100:.2f}%")

P(+) = 0.1085 = 10.85%

Only about one in ten tested people will test positive. Most of those positives are false alarms — not because the test is bad, but because the disease is rare.

✓ Phase 3 complete. P(+) = 0.1085 — the marginal probability of a positive test.

Phase 4: The Posterior — Compute What You Actually Want

Goal: Given a positive test, compute the actual probability of having the disease.

Now we combine the prior, likelihood, and evidence into the answer:

$P(D\mid +)=\frac{P(+\mid D)\times P(D)}{P(+)}=\frac{0.95\times 0.01}{0.1085}=\frac{0.0095}{0.1085}=0.0876$

Let's trace this with population counts. Imagine 10,000 people screened:

Every blue box in the tree — the initial branch into disease/healthy, then into test results — tells the same story. The final box highlights the 1,085 positive tests: only 95 are real.

P_D_given_pos = (P_pos_D * P_D) / P_pos
print(f"P(D|+)  = {P_D_given_pos:.4f} = {P_D_given_pos*100:.2f}%")
print(f"P(¬D|+) = {1-P_D_given_pos:.4f} = {(1-P_D_given_pos)*100:.2f}%")

P(D|+)  = 0.0876 = 8.76%
P(¬D|+) = 0.9124 = 91.24%

Even with a positive test, there's only an 8.76% chance of actually having the disease. The positive test raised the probability from 1% (prior) to 8.76% (posterior) — an 8.76× update. But 91.24% of positive tests are still false positives. This is not a flaw in the test; it's the mathematical consequence of a low base rate.

✓ Phase 4 complete. Posterior P(D|+) = 8.76% — a positive test does not mean you have the disease.

Trace Table — Full Calculation Path

Effect of Changing the Prior

The posterior depends heavily on the prior. A test with the same sensitivity (95%) and specificity (90%) gives dramatically different posteriors depending on disease prevalence:

priors = [0.001, 0.01, 0.05, 0.10, 0.50]
print(f"{'Prior P(D)':>12} {'P(+)':>8} {'Posterior P(D|+)':>18}")
for P_D in priors:
    P_pos = 0.95*P_D + 0.10*(1-P_D)
    posterior = (0.95 * P_D) / P_pos
    print(f"{P_D:>12.3f} {P_pos:>8.4f} {posterior:>18.4f} ({posterior*100:.1f}%)")

Prior P(D)     P(+)  Posterior P(D|+)
       0.001   0.1009           0.0094 (0.9%)
       0.010   0.1085           0.0876 (8.8%)
       0.050   0.1425           0.3333 (33.3%)
       0.100   0.1750           0.5143 (51.4%)
       0.500   0.5250           0.9048 (90.5%)

At prior = 0.1%, a positive test barely moves the needle (0.9% posterior). At prior = 50%, a positive test is almost definitive (90.5%). The test never changed — the prior does the heavy lifting. This is the most important lesson: the prior dominates when the base rate is extreme. A test that sounds impressive (95% sensitive, 90% specific) is practically useless for screening a rare disease.

The hyperparameter here — the prior — behaves like a regularizer in machine learning. A high prior (common condition) means the test result is reliable. A low prior (rare condition) means the test can't save you from the base rate. When you're building a classifier for rare events, this same math applies: your model's precision will always be constrained by the prevalence in your training data.

Connecting to ML: Generative vs Discriminative

Bayes theorem is the foundation of generative classifiers: models that explicitly learn $P(x\mid y)$ (how features are distributed within each class) and $P(y)$ (class prior), then infer the class label via:

$P(y\mid x)=\frac{P(x\mid y)\times P(y)}{P(x)}$

Naive Bayes, Linear Discriminant Analysis, and Hidden Markov Models are all generative.

Discriminative classifiers (logistic regression, SVM, random forests) skip the generative model and directly learn $P(y\mid x)$ from training data. They don't need to know how features were generated — they just need to learn the decision boundary.

Generative models require stronger assumptions but generalize better with less data. Discriminative models are more flexible but need more samples to estimate the decision boundary well.

Related Concepts

Backward: Bayes theorem is the mathematical backbone of probability theory — it appears in every generative model. The evidence $P(+)$ computed via the law of total probability is the denominator that normalizes the posterior. Without it, posteriors from different hypotheses can't be compared. This setup is identical to how logistic regression computes $P(y=1\mid x)$ using the log-odds ratio, just with a direct modeling approach instead of an inverse one.

Forward: Naive Bayes (post 02) applies Bayes theorem to classification by modeling $P(x\mid y)$ with a specific distribution (Gaussian, Multinomial, or Bernoulli). The disease-test example uses the same posterior computation but with binary features — once you understand the prior→likelihood→posterior chain here, the classifier follows immediately.

Honest Limitations

Here's the thing about Bayes theorem — I've seen teams apply it blindly and get burned, every time for the same few reasons. The first is rare events: when prevalence is 0.1%, a positive test gives 0.9% posterior. This feels impossibly low and it's deeply counterintuitive. The error isn't in the math — it's that we intuitively conflate test accuracy with predictive value. I've watched product teams build "high accuracy" classifiers for rare-event detection (fraud, equipment failure, disease screening) and then panic when the precision is below 5%. The math was correct all along; the prior was doing its work.

Second, the prior dominates at extreme prevalences. If you're screening for something that affects 0.1% of the population, almost no test can give you reliable positive predictions. The better approach is to pre-filter the population (use risk factors, family history, symptoms) to shift the prior before testing. In ML terms, the prior acts as a regularizer: you can't engineer your way around a bad base rate with a better model.

Third, Bayes theorem assumes test results are independent given disease status. In practice, false positives cluster — a faulty batch of test kits, a technician who misreads certain values, a patient who took the test incorrectly. When conditional independence is violated, the computed posterior underestimates uncertainty in predictable ways. This is exactly the same assumption Naive Bayes makes about features, and it fails in the same way: correlated errors inflate confidence.

Test Your Understanding

1. The posterior $P(D\mid +)=8.76\%$. After a second independent positive test, what is the new posterior? Use the first posterior (8.76%) as the new prior for the second test — compute $P(D\mid ++)$.

2. Test specificity increases from 90% to 99% (false positive rate drops from 10% to 1%). Recompute $P(D\mid +)$ for the original disease prevalence of 1%. How does this compare to 8.76%?

3. In the population of 10,000: 95 true positives and 990 false positives. If you screen only a high-risk sub-population where prevalence is 10% (instead of 1%), how many true and false positives would you expect? What is $P(D\mid +)$ in this sub-population?

4. The formula $P(y\mid x)\propto P(x\mid y)\times P(y)$ assumes independent class posterior computation. If we have 3 classes and compute the un-normalized posterior for each, how do we normalize to get probabilities summing to 1?

5. Generative classifiers model $P(x\mid y)$, which requires specifying how features are distributed. Naive Bayes assumes Gaussian or Multinomial distributions. What goes wrong if the actual feature distribution is heavily skewed and you use a Gaussian assumption?