Variants of Naive Bayes

You're building a spam filter. Every email that lands in the server needs a call in milliseconds — spam or inbox. You have a set of hand-labeled examples, but you can't afford to run an iterative optimization on each incoming message. You need something trained in a single pass through the data.

Now here's the catch: the emails contain words, and you need to decide which variant of Naive Bayes to use. Should you count how many times each word appears? Should you just note whether a word is present or absent? Or should you treat some continuous measurement — like the email's length in characters — as a feature? Each choice leads to a different likelihood model, and picking the wrong one silently degrades your classifier.

This is where understanding the variants of Naive Bayes matters. All three start from the same place — Bayes theorem with the naive independence assumption — but they differ in how they model P(x|y), the distribution of features within each class. If your features are word counts, use Multinomial NB. If they're binary presence indicators, use Bernoulli NB. If they're continuous measurements, use Gaussian NB. The math is nearly identical; the likelihood function is what changes.

The Naive Assumption — One Formula, Three Likelihoods

Let's start with what all three share. Naive Bayes classifies by computing the unnormalized posterior:

$P(y\mid x)\propto P(y)\times {\prod }_{j=1}^{p}P(x_j\mid y)$

The product form comes from assuming conditional independence: given the class, each feature is independent of all others:

$P(x_1,x_2,\dots \mid y)=P(x_1\mid y)\times P(x_2\mid y)\times \cdots$

This assumption is almost certainly violated in real data — "free" and "money" co-occur in spam far more than independence would predict. But the class prediction (argmax over $y$) is often correct even when individual probabilities are off. What changes between variants is the shape of $P(x_j\mid y)$.

Anchor dataset: Email spam classification. Four training emails, one test email.

train_emails = [
    {"free": 3, "money": 2, "hello": 0, "meeting": 1},  # spam
    {"free": 0, "money": 0, "hello": 2, "meeting": 3},  # ham
    {"free": 2, "money": 1, "hello": 0, "meeting": 0},  # spam
    {"free": 0, "money": 0, "hello": 1, "meeting": 2},  # ham
]
train_labels = [1, 0, 1, 0]

test_email = {"free": 1, "money": 1, "hello": 0, "meeting": 0}

The Plan — Three Likelihood Models

We'll walk through all three variants on the same classification problem. Each variant uses a different model for $P(x\mid y)$: counts, binary presence, or continuous density. After the three variants, we'll look at Laplace smoothing — the fix for the zero-count problem that all count-based variants share.

Phase 1: Multinomial Naive Bayes — Word Counts

Goal: Classify the test email using word frequency counts as features.

Multinomial NB handles features that are counts — how many times each word appears. The likelihood is the probability of drawing this bag of words from the class's word distribution.

Step 1 — Class priors. Two spam emails and two ham emails out of four total:

$P(\text{spam})=2/4=0.5P(\text{ham})=2/4=0.5$

Step 2 — Word likelihoods with Laplace smoothing. Count total word occurrences per class. Vocabulary $V=\{\text{free},\text{money},\text{hello},\text{meeting}\}$, $|V|=4$.

With Laplace smoothing ($\alpha =1$):

$P(\text{word}\mid \text{class})=\frac{\text{count}(\text{word},\text{class})+1}{\text{total words in class}+|V|}$

Spam emails (indices 0 and 2): free = 3+2 = 5, money = 2+1 = 3, hello = 0, meeting = 1+0 = 1. Total words = 9. With smoothing: denominator = 9 + 4 = 13.

Ham emails (indices 1 and 3): free = 0, money = 0, hello = 2+1 = 3, meeting = 3+2 = 5. Total words = 8. With smoothing: denominator = 8 + 4 = 12.

Step 3 — Classify the test email. Test email has free:1, money:1, hello:0, meeting:0. For Multinomial NB:

$P(\text{email}\mid y)={\prod }_{j}P({\text{word}}_j\mid y{)}^{{\text{count}}_j}$

Words with count zero contribute $P(\text{word}\mid y{)}^0=1$ — absent words are simply ignored.

$P(\text{email}\mid \text{spam})=0.462^1\times 0.308^1\times 0.077^0\times 0.154^0=0.462\times 0.308=0.142$

$P(\text{email}\mid \text{ham})=0.083^1\times 0.083^1\times 0.333^0\times 0.500^0=0.083\times 0.083=0.007$

Unnormalized posteriors:

$P(\text{spam}\mid \text{email})\propto 0.5\times 0.142=0.071$ $P(\text{ham}\mid \text{email})\propto 0.5\times 0.007=0.0035$

Normalize by dividing by their sum:

$P(\text{spam})=\frac{0.071}{0.071+0.0035}=0.953\Rightarrow \text{SPAM}$

The email mentions "free" and "money" — two words almost exclusive to spam in this training set. The result matches intuition.

The blue bars (spam) are dominated by "free" and "money"; the red bars (ham) by "hello" and "meeting". The test email only activates the left two words, giving spam a massive likelihood advantage.

✓ Phase 1 complete. Multinomial NB classifies the test email as spam with 95.3% confidence, based on word counts.

Phase 2: Bernoulli Naive Bayes — Binary Presence/Absence

Goal: Classify the test email using binary word presence/absence instead of counts.

Bernoulli NB converts word counts to binary values: 1 if the word appears at least once, 0 if it doesn't. This is NOT the same as Multinomial NB — here, absent words actively contribute to the likelihood rather than being ignored.

Step 1 — Binary conversion:

Step 2 — Word presence likelihoods. With Laplace smoothing ($\alpha =1$, $n_c=2$ per class):

$P(\text{word}=1\mid \text{class})=\frac{\text{count present in class}+1}{n_c+2}$

Step 3 — Classify the test email. Test email: free=1, money=1, hello=0, meeting=0.

The critical difference from Multinomial NB: Bernoulli NB uses both present and absent words. Absent words contribute $P(\text{word}=0\mid y)=1-P(\text{word}=1\mid y)$.

$P(\text{email}\mid \text{spam})=P(\text{free}=1)\times P(\text{money}=1)\times P(\text{hello}=0)\times P(\text{meeting}=0)$ $=0.75\times 0.75\times (1-0.25)\times (1-0.50)=0.75\times 0.75\times 0.75\times 0.50=0.211$

$P(\text{email}\mid \text{ham})=P(\text{free}=1)\times P(\text{money}=1)\times P(\text{hello}=0)\times P(\text{meeting}=0)$ $=0.25\times 0.25\times (1-0.75)\times (1-0.75)=0.25\times 0.25\times 0.25\times 0.25=0.0039$

Unnormalized posteriors:

$P(\text{spam}\mid \text{email})\propto 0.5\times 0.211=0.1055$ $P(\text{ham}\mid \text{email})\propto 0.5\times 0.0039=0.00195$

Normalized:

$P(\text{spam})=\frac{0.1055}{0.1055+0.00195}=0.982\Rightarrow \text{SPAM}$

The absent words (hello=0, meeting=0) actually increased confidence in spam here: ham emails always contain hello and meeting, so their absence is evidence against ham. Multinomial NB silently ignored them.

The green bars show the absent-word contributions — for the test email's missing "hello" and "meeting", the absence strongly favors spam over ham.

✓ Phase 2 complete. Bernoulli NB classifies the test email as spam with 98.2% confidence, using both presence and absence of words.

Phase 3: Gaussian Naive Bayes — Continuous Features

Goal: Classify a sample with continuous measurements using Gaussian density estimates.

For continuous features, we can't count word frequencies — we need a different likelihood model. Gaussian NB assumes each feature follows a normal distribution within each class:

$P(x_j\mid y)=\frac{1}{\sqrt{2\pi {\sigma }_{jy}^2}}\exp \left(-\frac{(x_j-{\mu }_{jy}{)}^2}{2{\sigma }_{jy}^2}\right)$

The parameters ${\mu }_{jy}$ and ${\sigma }_{jy}^2$ are the mean and variance of feature $j$ for samples in class $y$, estimated from training data.

The email spam dataset uses word counts, which aren't continuous. For this variant we switch to a dataset with real-valued features — the Iris dataset — to demonstrate Gaussian likelihoods:

# Iris setosa:     μ_sepal=5.01, σ_sepal=0.35
# Iris versicolor: μ_sepal=5.94, σ_sepal=0.51

For a test sample with sepal_length = 5.5:

$P(5.5\mid \text{Setosa})=\frac{1}{\sqrt{2\pi (0.35{)}^2}}\exp \left(-\frac{(5.5-5.01{)}^2}{2(0.35{)}^2}\right)=\frac{1}{0.877}\exp (-0.980)=0.421$

$P(5.5\mid \text{Versicolor})=\frac{1}{\sqrt{2\pi (0.51{)}^2}}\exp \left(-\frac{(5.5-5.94{)}^2}{2(0.51{)}^2}\right)=\frac{1}{1.277}\exp (-0.372)=0.523$

Sepal length alone slightly favors Versicolor. In a full classifier, the petal_length likelihood (which separates Setosa sharply) would be combined via the same product — the joint posterior correctly classifies most Iris samples.

import numpy as np
from scipy.stats import norm

mu_setosa, sigma_setosa = 5.01, 0.35
mu_versi, sigma_versi = 5.94, 0.51
x = 5.5

P_setosa = norm.pdf(x, mu_setosa, sigma_setosa)
P_versi  = norm.pdf(x, mu_versi, sigma_versi)
print(f"P(5.5 | Setosa)     = {P_setosa:.4f}")
print(f"P(5.5 | Versicolor) = {P_versi:.4f}")

P(5.5 | Setosa)     = 0.4213
P(5.5 | Versicolor) = 0.5235

The Gaussian PDF gives smooth, continuous likelihood estimates. Combine this over all features, multiply by the class prior, and you get a probabilistic classification.

✓ Phase 3 complete. Gaussian NB computes likelihoods from a normal density — sepal_length=5.5 favors Versicolor (0.524 vs 0.421).

Phase 4: Laplace Smoothing — The Zero Probability Problem

Goal: Prevent unseen words from zeroing out the entire posterior.

Without smoothing, if the word "bitcoin" never appears in spam training emails:

$P(\text{bitcoin}\mid \text{spam})=0/1000=0$

Any email containing "bitcoin" gets $P(\text{spam}\mid \text{email})=P(\text{spam})\times 0\times \dots =0$, regardless of every other word. One unseen word destroys the classification — the zero probability kills the entire product.

Laplace smoothing fixes this by adding a small pseudocount $\alpha$ to every word count:

$P(\text{word}\mid \text{class})=\frac{\text{count}(\text{word},\text{class})+\alpha }{\text{total words in class}+\alpha \times |V|}$

$\alpha =1$ gives add-1 (uniform) smoothing — every unseen word gets the same small probability. $\alpha <1$ is less aggressive, preserving more discriminative signal for common words. $\alpha$ is a hyperparameter: cross-validate it like any other.

✓ Phase 4 complete. Laplace smoothing ensures zero-count words don't zero the posterior.

Trace Table — Variant Comparison on the Test Email

Hyperparameter Sensitivity — The Effect of $\alpha$

The smoothing parameter $\alpha$ controls how much probability mass is reserved for unseen words. Too small, and rare words dominate. Too large, and all words wash out toward uniform:

from functools import reduce

train_emails = [
    {"free": 3, "money": 2, "hello": 0, "meeting": 1},
    {"free": 0, "money": 0, "hello": 2, "meeting": 3},
    {"free": 2, "money": 1, "hello": 0, "meeting": 0},
    {"free": 0, "money": 0, "hello": 1, "meeting": 2},
]
train_labels = [1, 0, 1, 0]
test_email   = {"free": 1, "money": 1, "hello": 0, "meeting": 0}

def multinomial_nb_score(test, train_emails, train_labels, alpha):
    vocab = ["free", "money", "hello", "meeting"]
    classes = [0, 1]
    scores = {}
    for c in classes:
        email_indices = [i for i, lbl in enumerate(train_labels) if lbl == c]
        total_words = sum(
            train_emails[i].get(w, 0)
            for i in email_indices for w in vocab
        )
        denom = total_words + alpha * len(vocab)
        prior = len(email_indices) / len(train_labels)
        likelihood = 1.0
        for w in vocab:
            count = sum(train_emails[i].get(w, 0) for i in email_indices)
            p = (count + alpha) / denom
            likelihood *= p ** test.get(w, 0)
        scores[c] = prior * likelihood
    return {c: s / sum(scores.values()) for c, s in scores.items()}

alphas = [0.001, 0.01, 0.1, 0.5, 1.0, 2.0, 5.0]
print(f"{'alpha':>8} {'P(spam)':>10} {'P(ham)':>10} {'Prediction':>12}")
for a in alphas:
    probs = multinomial_nb_score(test_email, train_emails, train_labels, a)
    pred  = "SPAM" if probs[1] > probs[0] else "HAM"
    print(f"{a:>8.3f} {probs[1]:>10.4f} {probs[0]:>10.4f} {pred:>12}")

alpha    P(spam)      P(ham)  Prediction
   0.001     0.9999     0.0001        SPAM
   0.010     0.9962     0.0038        SPAM
   0.100     0.9743     0.0257        SPAM
   0.500     0.9535     0.0465        SPAM
   1.000     0.9526     0.0474        SPAM
   2.000     0.9314     0.0686        SPAM
   5.000     0.8571     0.1429        SPAM

At $\alpha =0.001$, the model is maximally confident (0.9999) — rare words get extreme probabilities, making one class dominate. As $\alpha$ grows, posteriors soften toward 50/50. All values predict SPAM on this test email because the training data is unambiguous, but the confidence margin narrows from 99.99% to 85.71%. On a larger vocabulary with more unseen words, this narrowing would be even more dramatic — and at some point a large $\alpha$ can flip the prediction by diluting discriminative signal.

Multinomial vs Bernoulli NB — When to Use Which

The absent-word difference is the practical distinction: Bernoulli penalizes words that are characteristic of the class but don't appear, which can hurt or help depending on the task.

Related Concepts

Backward: The three variants build directly on Bayes theorem and conditional independence from the previous post in this series. Choosing the right variant depends entirely on your feature type — the same independence assumption underlies all three, just with different likelihood models. Understanding the log-space view ($\log P(y\mid x)=\log P(y)+{\sum }_j{x}_j\log P(x_j\mid y)$) reveals that Naive Bayes is a linear classifier in log-space — the same functional form as logistic regression, but with weights set analytically from counts rather than learned via gradient descent.

Forward: Multinomial NB is the entry point to the broader bag-of-words pipeline (TF-IDF, BM25, Complement NB for imbalanced classes). Gaussian NB generalizes to Gaussian discriminant analysis, where relaxing the diagonal covariance assumption leads to quadratic discriminant analysis. Understanding how each variant handles absent features prepares you for Laplacian correction in other probabilistic models and for smoothing in language models.

Honest Limitations

Here's what I've learned from watching Naive Bayes fail in production. First, the independence assumption — it's wrong in every real dataset, and the problem isn't just that the probabilities are off. When features are correlated, they all "vote" in the same direction, and the product double-counts the redundant evidence. The result is posteriors that cluster near 0 and 1 even when the true posterior is 0.6. I've seen teams deploy spam filters that report 99.9% confidence on every decision, then get confused when precision at that threshold doesn't match expectations. The model was overconfident because it treated "free" and "free!!!" as independent pieces of evidence.

Second, Gaussian NB breaks quietly on non-Gaussian features. Word counts, income, latency measurements — these are often heavy-tailed or multimodal within a class. The Gaussian fit produces poor density estimates, and the resulting likelihoods distort the posterior. I've seen this happen with sensor data where a feature has two clusters within the same class. The Gaussian averages them into one wide blob, giving every sample near-identical likelihoods and making the classifier useless.

Third, all three variants share a limitation that's baked into the bag-of-words representation: they can't capture feature interactions. "Not spam" looks like spam to a bag-of-words model because it contains the token "spam." Negation, collocations, and idiomatic phrases are invisible without n-gram features or more sophisticated representations. If your problem depends on word order or interaction effects, Naive Bayes will hit a ceiling regardless of how much data you feed it.

Test Your Understanding

1. Multinomial NB ignores absent words ($P^0=1$). If you add a fifth word "bitcoin" to the vocabulary but it appears zero times in training, does Laplace smoothing change the classification of the test email {free:1, money:1}? Why or why not?

2. In Bernoulli NB, the absent word "meeting" contributed $(1-P(\text{meeting}=1\mid \text{spam}))=1-0.50=0.50$ to the spam likelihood. In Multinomial NB, "meeting" contributes $0.154^0=1$. Which treatment is more conservative (less confident) about spam, and why?

3. Gaussian NB assumes features are normally distributed within each class. The Iris dataset has sepal_length near-Gaussian, but word counts in text are typically Poisson or power-law distributed. What happens to the likelihood estimates if you apply Gaussian NB to word count features?

4. Laplace smoothing with $\alpha =1$ adds the same count to every word. If your vocabulary has 50,000 words but only 500 are ever seen in spam, how does $\alpha =1$ vs $\alpha =0.01$ affect rare vs common word probabilities?

5. Both Multinomial NB and Bernoulli NB classify the test email as spam, but with different confidences (0.953 vs 0.982). These probabilities are not calibrated — they overstate confidence. Why does the independence assumption inflate posterior probabilities toward 0 and 1?