Chi-Square Test

Your model makes predictions in three categories: Positive, Negative, and Neutral. You have two versions of the model — call them Model A and Model B. Are the prediction distributions the same, or does one version systematically differ in how it assigns categories? Continuous measurements call for t-tests. Categorical counts call for the chi-square test.

The chi-square test asks: do observed counts match expected counts under a hypothesis? It handles the question of whether two categorical variables are independent, whether distributions differ across groups, and whether data follows a specified distribution. Any time you have counts in categories and a question about whether the pattern is random, chi-square is the natural tool.

The Dataset

You deploy both model versions on 200 held-out inputs (100 per model) and record the predicted category:

Are Model A and Model B distributing predictions the same way across categories?

The Basic Idea

The chi-square statistic measures how far observed frequencies are from expected frequencies:

${\chi }^2={\sum }_{cells}\frac{(O-E{)}^2}{E}$

where $O$ is the observed count and $E$ is the expected count under $H_0$.

Large ${\chi }^2$ means large discrepancy — evidence against $H_0$. Under $H_0$, this statistic follows a chi-square distribution with degrees of freedom determined by the table structure.

Hypotheses

$H_0$: Model A and Model B produce the same prediction distribution across categories (the variables "model version" and "predicted category" are independent)

$H_1$: The prediction distributions differ

Significance level: $\alpha =0.05$

Phase 1: Expected Frequencies

Under independence, the expected count for each cell is:

$E_{ij}=\frac{({\text{Row Total}}_i)\times ({\text{Column Total}}_j)}{\text{Grand Total}}$

Computing each cell:

$E_{A,Pos}=\frac{100\times 97}{200}=48.5$ $E_{A,Neg}=\frac{100\times 66}{200}=33.0$ $E_{A,Neu}=\frac{100\times 37}{200}=18.5$ $E_{B,Pos}=\frac{100\times 97}{200}=48.5$ $E_{B,Neg}=\frac{100\times 66}{200}=33.0$ $E_{B,Neu}=\frac{100\times 37}{200}=18.5$

All expected frequencies exceed 5, so the chi-square approximation is valid.

Phase 2: $(O-E{)}^2/E$ Per Cell

${\chi }^2=0.871+0.758+0.122+0.871+0.758+0.122=3.502$

Phase 3: Degrees of Freedom and Decision

$df=(r-1)(c-1)=(2-1)(3-1)=1\times 2=2$

where $r=2$ rows (models) and $c=3$ columns (categories).

Critical value: ${\chi }_{0.05,2}^2=5.991$

Decision: $3.502<5.991$ — fail to reject $H_0$.

The data is consistent with Model A and Model B producing the same prediction distribution. The observed differences in counts are within normal random variation.

Effect Size: Cramér's V

Statistical significance does not equal practical importance. Cramér's V normalizes the chi-square for sample size and table dimensions:

$V=\sqrt{\frac{{\chi }^2}{n\times \min (r-1,c-1)}}=\sqrt{\frac{3.502}{200\times \min (1,2)}}=\sqrt{\frac{3.502}{200}}=\sqrt{0.01751}=0.132$

$V=0.132$ is a small effect — consistent with the non-significant chi-square. Even if the test had achieved significance with more data, the practical association is modest.

Fisher's Exact Test: When Expected Counts Are Small

If any expected cell count is below 5, the chi-square approximation is unreliable. Fisher's exact test computes the exact probability of the observed table (and more extreme tables) under $H_0$, without approximation. Use it when:

• Any $E_{ij}<5$
• Sample size is small overall
• The table is 2×2

import numpy as np
from scipy import stats

# Model prediction contingency table
contingency_table = np.array([
    [42, 38, 20],   # Model A: Positive, Negative, Neutral
    [55, 28, 17]    # Model B: Positive, Negative, Neutral
])

chi2, p, dof, expected = stats.chi2_contingency(contingency_table)

print(f"Chi-square statistic: {chi2:.4f}")
print(f"p-value: {p:.4f}")
print(f"Degrees of freedom: {dof}")
print(f"\nExpected frequencies:")
print(expected)

n = contingency_table.sum()
min_dim = min(contingency_table.shape) - 1
cramers_v = np.sqrt(chi2 / (n * min_dim))
print(f"\nCramer's V: {cramers_v:.4f}")

# Standardized residuals (which cells drive the chi-square most?)
residuals = (contingency_table - expected) / np.sqrt(expected)
print("\nStandardized residuals:")
print(residuals)

# Fisher's exact test (2x2 only — use if expected cells are small)
small_table = np.array([[42, 58], [55, 45]])
odds_ratio, p_fisher = stats.fisher_exact(small_table)
print(f"\nFisher's exact test (2x2 subset, Pos vs rest): p={p_fisher:.4f}")

The Assumptions

1. Categorical data: Counts in categories, not measurements
2. Independent observations: Each prediction belongs to exactly one cell
3. Adequate expected frequencies: At least 80% of cells should have $E\ge 5$, no cell $E<1$

If expected counts are too small: combine categories (e.g., merge Neutral into Negative/Positive), increase sample size, or use Fisher's exact test.

Three Flavors

Goodness of Fit: One variable. Does this model's prediction distribution match a theoretical distribution (e.g., uniform across classes)?

Test of Independence: Two variables, contingency table. Are model version and prediction category independent?

Homogeneity: Are prediction distributions identical across multiple model versions? (Same mechanics as independence, different question.)

Related Concepts

The chi-square test extends hypothesis testing (post 3) to categorical outcomes, where t-tests and Z-tests do not apply. It uses the chi-square distribution, which arises as the sum of squared standard normals — connected to the CLT (post 1) via the Normal approximation for counts. The goodness-of-fit test (post 13) applies the same statistic to single-variable questions like "does my model's output distribution match the true label distribution?" ANOVA (post 14) handles the continuous analog: comparing means across multiple groups instead of comparing count distributions.

Honest Limitations

The chi-square test detects association but says nothing about direction, magnitude, or cause. A significant chi-square means the prediction distributions differ — it does not tell you which category is causing the difference. Use standardized residuals (shown in the output above) to identify which cells contribute most. Also, chi-square only applies to counts — not proportions, means, or scores. Treating percentages or rates as counts inflates the test statistic and produces misleading results.

Test Your Understanding

1. You add a third model (Model C) with counts [48, 35, 17] in the same three categories. Recompute the expected frequencies for the 3×3 table and the new degrees of freedom.
2. A small experiment has only 30 predictions per model, resulting in expected cell counts of [14.5, 9.9, 5.6]. Can you safely use the chi-square test? What would you do instead?
3. Cramér's V = 0.132 for the comparison above. If you doubled the sample to 400 total predictions with the same proportions, would V change? Would the chi-square statistic change?
4. The standardized residual for cell (Model A, Positive) is -0.934. What does this tell you about how Model A's positive predictions compare to what independence predicts?
5. Two models predict binary labels (Positive vs Not-Positive) with counts [[42, 58], [55, 45]]. Compute the 2×2 chi-square by hand and verify it equals the sum of $(O-E{)}^2/E$ across all four cells.