Conditional Probability

The prosecutor's fallacy: P(DNA match | innocent) might be 0.0001. A prosecutor argues "the probability of innocence is 0.0001." Wrong — they're computing P(innocent | DNA match), which requires knowing the base rate of innocence in the suspect population. The two probabilities are very different quantities. Confusing them can send innocent people to jail.

ML evaluation version: P(predicted positive | fraud) = recall = 0.85. This does not mean 85% of positive predictions are fraud. That's precision = 0.63. Always name which direction you're conditioning.

Chain Rule

The definition P(A|B) = P(A∩B)/P(B) rearranges to:

This is the chain rule for two events. For three events, apply iteratively:

Derive step by step:

1. P(A∩B∩C) = P(A | B∩C) × P(B∩C) [from definition, conditioning on B∩C]
2. P(B∩C) = P(B|C) × P(C) [from definition, conditioning on C]
3. Substitute: P(A∩B∩C) = P(A|B∩C) × P(B|C) × P(C)

Verification on anchor:

Check: 85/1000 = 0.085 ✓

Leaves sum to 1: 0.085+0.015+0.050+0.850 = 1.000 ✓

Independence: Formal Definition

Events A and B are independent if and only if:

Equivalently: P(A ∩ B) = P(A) × P(B) — the joint probability factorizes.

Test on anchor: Is fraud independent of the model's prediction?

Correct — a useful model correlates its predictions with truth. A random classifier would have P(fraud | PP) = P(fraud) = 0.10 (knowing the prediction gives no information). That would be a useless, maximally uninformative model.

Conditional independence: Two events can be dependent unconditionally but independent given a third event. Two model features might be correlated unconditionally, but when you condition on the true class label, the correlation vanishes — the features share no information beyond what the class already explains. This is the assumption Naive Bayes exploits:

Each feature's likelihood is estimated separately under the class label. The "naive" label reflects that this conditional independence assumption is usually wrong — but the posterior ranking is often still correct.

Law of Total Probability

When you cannot compute P(B) directly, decompose it through a partition of the sample space. If {A₁, A₂, ..., Aₖ} partitions Ω (mutually exclusive, exhaustive):

Derivation on anchor: Compute P(predicted positive) without using the column total directly.

The partition: fraud (A₁) and legitimate (A₂).

Check: 135/1000 = 0.135 ✓

This is how you compute a marginal probability when you only know the conditional probabilities and the partition weights.

Connection to Bayes' Theorem

Bayes' theorem follows directly from the definition of conditional probability applied in both directions:

Applied once on the anchor using the law of total probability for the denominator:

Matches the direct calculation ✓. The full treatment — prior updating, sequential Bayesian inference, base rate neglect — is in the Bayes' theorem post. Here the key point is that Bayes' theorem is a consequence of the conditional probability definition, not a new axiom.

ML Applications

Precision and recall as conditional probabilities:

Every confusion matrix metric has a conditional probability interpretation. When you report model performance, you're reporting conditional probabilities — always name which direction you're conditioning. Reporting recall when your stakeholder needs precision is a technically correct but operationally misleading answer.

Naive Bayes and class-conditional likelihoods:

A Naive Bayes classifier estimates P(features | class) × P(class) for each class, then takes the argmax. Each term P(xᵢ | y) is a conditional probability estimated from training data: "given the class is y, what is the distribution of feature xᵢ?" The classifier exploits these conditional distributions plus the chain rule to assign class probabilities without ever needing to estimate the joint distribution of all features simultaneously.

Model calibration:

A well-calibrated model satisfies P(y=1 | score=s) ≈ s for all score values s. If the model outputs score=0.9 but P(y=1 | score=0.9) = 0.40 in reality, the model is overconfident — it says "90% confident" but only 40% of those predictions are correct. Calibration is precisely a conditional probability statement about the model's score output.

TP, FP, FN, TN = 85, 50, 15, 850
n = TP + FP + FN + TN

precision   = TP / (TP + FP)
recall      = TP / (TP + FN)
for_rate    = FN / (FN + TN)
specificity = TN / (TN + FP)

print(f"Precision    P(fraud|PP)   = {precision:.3f}")
print(f"Recall       P(PP|fraud)   = {recall:.3f}")
print(f"Asymmetry:   {precision:.3f} ≠ {recall:.3f}")
print()
print(f"FOM rate     P(fraud|PN)   = {for_rate:.3f}")
print(f"Specificity  P(PN|legit)   = {specificity:.3f}")
print()
# Law of total probability
p_fraud = (TP + FN) / n
p_legit = (FP + TN) / n
p_pp_given_fraud = TP / (TP + FN)
p_pp_given_legit = FP / (FP + TN)
p_pp = p_pp_given_fraud * p_fraud + p_pp_given_legit * p_legit
print(f"P(PP) via total probability = {p_pp:.3f}  [direct: {(TP+FP)/n:.3f}]")

Related Concepts

Conditional probability is the foundation for Bayes' theorem (updating beliefs with evidence), the chain rule of probability (decomposing joint distributions), and Markov chains (where state transitions are conditional on the current state). In ML, it underlies precision, recall, Naive Bayes, model calibration, and every use of posterior probabilities. Independence testing — checking whether P(A|B) = P(A) — connects to feature selection and mutual information.

Honest Limitations

The conditional probability formula P(A|B) = P(A∩B)/P(B) requires P(B) > 0. Conditioning on a zero-probability event is undefined. In practice this means: if the conditioning event is very rare (B has very few samples), the estimate of P(A|B) becomes high-variance and unreliable. With 5 instances of a rare event type in your dataset, P(A | rare event) computed from those 5 cases is a noisy estimate. This is the small denominator problem in conditional probability estimation, and it requires either smoothing, Bayesian priors, or more data.

Test Your Understanding

1. A spam filter achieves: TP=180, FP=20, FN=30, TN=770. Compute P(spam | predicted spam) and P(predicted spam | spam). Explain in one sentence why these are different.

2. Using the conditional probability formula, show algebraically why P(A|B) ≠ P(B|A) in general. Under what condition would they be equal?

3. The chain rule gives P(A∩B∩C) = P(A|B∩C) × P(B|C) × P(C). Use this to compute the probability that three independent sensor alarms all fire simultaneously, where each fires with probability 0.05 independently.

4. From the anchor confusion matrix, compute P(predicted positive) two ways: (a) directly from column totals, and (b) using the law of total probability. Verify they match.

5. A model for rare disease detection (prevalence = 1%) has sensitivity (recall) = 0.99 and specificity = 0.95. Compute the precision (positive predictive value) using Bayes' theorem. Why is the result surprising given the high sensitivity and specificity?