Variants of Naive Bayes

Bayes theorem gives the posterior $P(y\mid x)\propto P(y)\times P(x\mid y)$. To make this computable, you need a model for the likelihood $P(x\mid y)$ — how features are distributed given the class. Three different assumptions about this distribution give three different classifiers: Multinomial NB for count data, Bernoulli NB for binary presence/absence, and Gaussian NB for continuous measurements.

Anchor dataset: Email spam classification. Four training emails, one test email.

# Training data: word counts per email, label (1=spam, 0=ham)
train_emails = [
    {"free": 3, "money": 2, "hello": 0, "meeting": 1},  # spam
    {"free": 0, "money": 0, "hello": 2, "meeting": 3},  # ham
    {"free": 2, "money": 1, "hello": 0, "meeting": 0},  # spam
    {"free": 0, "money": 0, "hello": 1, "meeting": 2},  # ham
]
train_labels = [1, 0, 1, 0]

test_email = {"free": 1, "money": 1, "hello": 0, "meeting": 0}

The Naive Assumption

Naive Bayes classifies by computing:

$P(y\mid x)\propto P(y)\times {\prod }_{j=1}^{p}P(x_j\mid y)$

The product form comes from assuming conditional independence: given the class, each feature is independent of all others. $P(x_1,x_2,\dots \mid y)=P(x_1\mid y)\times P(x_2\mid y)\times \cdots$

This is almost certainly violated in real data — "free" and "money" co-occur in spam far more than independence would predict. But the class prediction (argmax over $y$) is often correct even when individual probabilities are off. Naive Bayes can be thought of as a linear classifier in log-space:

$\log P(y\mid x)=\log P(y)+{\sum }_j{x}_j\log P(x_j\mid y)+\text{const}$

This is linear in features $x_j$ — the same functional form as logistic regression, but with parameters set analytically from counts rather than gradient descent.

Variant 1: Multinomial Naive Bayes

Multinomial NB treats each feature as a word count. The likelihood $P(x\mid y)$ is the probability of observing this bag-of-words given the class.

Step 1: Class priors

$P(\text{spam})=2/4=0.5,P(\text{ham})=2/4=0.5$

Step 2: Word likelihoods with Laplace smoothing

Count total word occurrences per class. Vocabulary $V=\{$free, money, hello, meeting$\}$, $|V|=4$.

With Laplace smoothing ($\alpha =1$): $P(\text{word}\mid \text{class})=\frac{\text{count}(\text{word, class})+1}{\text{total words in class}+|V|}$

Spam emails (emails 1 and 3): free=3+2=5, money=2+1=3, hello=0, meeting=1+0=1. Total = 9. With smoothing: denominator = 9 + 4 = 13.

Ham denominator: total words = 0+0+2+3+0+0+1+2 = 8. With smoothing: 8 + 4 = 12.

Step 3: Classify the test email {free:1, money:1, hello:0, meeting:0}

For Multinomial NB: $P(\text{email}\mid y)={\prod }_{j}P({\text{word}}_j\mid y{)}^{{\text{count}}_j}$

Zero-count words contribute $P(\text{word}\mid y{)}^0=1$ — absent words are ignored.

$P(\text{email}\mid \text{spam})=0.462^1\times 0.308^1\times 0.077^0\times 0.154^0=0.462\times 0.308=0.142$

$P(\text{email}\mid \text{ham})=0.083^1\times 0.083^1\times 0.333^0\times 0.500^0=0.083\times 0.083=0.007$

Unnormalized posteriors: $P(\text{spam}\mid \text{email})\propto 0.5\times 0.142=0.071$ $P(\text{ham}\mid \text{email})\propto 0.5\times 0.007=0.0035$

Normalized: $P(\text{spam})=0.071/(0.071+0.0035)=0.953$ → SPAM

The email mentions "free" and "money" — two words almost exclusive to spam in this training set. The result matches intuition.

Variant 2: Bernoulli Naive Bayes

Bernoulli NB converts word counts to binary presence/absence. It treats each word as a separate Bernoulli variable: 1 if present, 0 if absent.

Step 1: Binary conversion

Step 2: Word presence likelihoods (2 spam emails, 2 ham emails; $\alpha =1$, denominator $n_c+2$)

$P(\text{word}=1\mid \text{class})=\frac{\text{count present in class}+1}{n_c+2}$

Step 3: Classify test email {free:1, money:1, hello:0, meeting:0}

The critical difference: Bernoulli NB uses both present and absent words. Absent words contribute $P(\text{word}=0\mid y)=1-P(\text{word}=1\mid y)$.

$P(\text{email}\mid \text{spam})=P(\text{free}=1)\times P(\text{money}=1)\times P(\text{hello}=0)\times P(\text{meeting}=0)$ $=0.75\times 0.75\times (1-0.25)\times (1-0.50)=0.75\times 0.75\times 0.75\times 0.50=0.211$

$P(\text{email}\mid \text{ham})=P(\text{free}=1)\times P(\text{money}=1)\times P(\text{hello}=0)\times P(\text{meeting}=0)$ $=0.25\times 0.25\times (1-0.75)\times (1-0.75)=0.25\times 0.25\times 0.25\times 0.25=0.0039$

Unnormalized posteriors:

• $P(\text{spam})\propto 0.5\times 0.211=0.1055$
• $P(\text{ham})\propto 0.5\times 0.0039=0.00195$

Normalized: $P(\text{spam})=0.1055/(0.1055+0.00195)=0.982$ → SPAM

The absent words (hello=0, meeting=0) actually increased confidence in spam here: ham emails always contain hello and meeting, so their absence is evidence against ham. Multinomial NB silently ignored them.

Variant 3: Gaussian Naive Bayes

For continuous features, assume each feature follows a Gaussian distribution within each class:

$P(x_j\mid y)=\frac{1}{\sqrt{2\pi {\sigma }_{jy}^2}}\exp \left(-\frac{(x_j-{\mu }_{jy}{)}^2}{2{\sigma }_{jy}^2}\right)$

Parameters ${\mu }_{jy}$ and ${\sigma }_{jy}^2$ are estimated from training data (mean and variance of feature $j$ in class $y$).

Iris example — sepal length:

Approximate parameters from the Iris dataset:

• Setosa: $\mu =5.01$, $\sigma =0.35$
• Versicolor: $\mu =5.94$, $\sigma =0.51$

For a test sample with sepal_length = 5.5:

$P(5.5\mid \text{Setosa})=\frac{1}{\sqrt{2\pi (0.35{)}^2}}\exp \left(-\frac{(5.5-5.01{)}^2}{2(0.35{)}^2}\right)=\frac{1}{0.877}\exp (-0.980)=0.421$

$P(5.5\mid \text{Versicolor})=\frac{1}{\sqrt{2\pi (0.51{)}^2}}\exp \left(-\frac{(5.5-5.94{)}^2}{2(0.51{)}^2}\right)=\frac{1}{1.277}\exp (-0.372)=0.523$

Sepal length alone slightly favors Versicolor. Combined with petal_length likelihood (which separates Setosa sharply), the joint posterior correctly classifies most Iris samples.

Laplace Smoothing — The Zero Probability Problem

Without smoothing: if "bitcoin" never appears in 1,000 spam training emails:

$P(\text{bitcoin}\mid \text{spam})=0/1000=0$

Any email containing "bitcoin" gets $P(\text{spam}\mid \text{email})=P(\text{spam})\times 0\times \dots =0$, regardless of every other word. One unseen word destroys the classification.

Laplace smoothing adds $\alpha$ to every word count:

$P(\text{word}\mid \text{class})=\frac{\text{count}(\text{word, class})+\alpha }{\text{total words in class}+\alpha \times |V|}$

$\alpha =1$: add-1 (uniform) smoothing. $\alpha <1$: smaller pseudocount — less aggressively uniform. The right $\alpha$ is a hyperparameter; cross-validate it (post 03).

Multinomial vs Bernoulli NB

The absent-word difference is the practical distinction: Bernoulli penalizes words that are characteristic of the class but don't appear, which can hurt or help depending on the task.

Variants Summary

Test Your Understanding

1. Multinomial NB ignores absent words ($P^0=1$). If you add a fifth word "bitcoin" to the vocabulary but it appears zero times in training, does Laplace smoothing change the classification of the test email {free:1, money:1}? Why or why not?

2. In Bernoulli NB, the absent word "meeting" contributed $(1-P(\text{meeting}=1\mid \text{spam}))=1-0.50=0.50$ to the spam likelihood. In Multinomial NB, "meeting" contributes $0.154^0=1$. Which treatment is more conservative (less confident) about spam, and why?

3. Gaussian NB assumes features are normally distributed within each class. The Iris dataset has sepal_length near-Gaussian, but word counts in text are typically Poisson or power-law distributed. What happens to the likelihood estimates if you apply Gaussian NB to word count features?

4. Laplace smoothing with $\alpha =1$ adds the same count to every word. If your vocabulary has 50,000 words but only 500 are ever seen in spam, how does $\alpha =1$ vs $\alpha =0.01$ affect rare vs common word probabilities?

5. Both Multinomial NB and Bernoulli NB classify the test email as spam, but with different confidences (0.953 vs 0.982). These probabilities are not calibrated — they overstate confidence. Why does the independence assumption inflate posterior probabilities toward 0 and 1?