ANOVA

You have three model architectures — a baseline CNN, a ResNet variant, and a transformer — each evaluated on 5 random seeds. You want to know whether they truly differ in performance. Running three pairwise t-tests would work mechanically, but with 5 architectures it becomes 10 tests, and with 10 it becomes 45. Each test carries a 5% false positive risk, so running many of them inflates your chance of declaring a spurious winner. ANOVA solves this by testing all groups simultaneously with a single test statistic.

The Core Idea: Signal to Noise

ANOVA does not compare means directly. It compares variance. Specifically:

$F=\frac{\text{Variance Between Groups}}{\text{Variance Within Groups}}=\frac{M{S}_{between}}{M{S}_{within}}$

Think of the F-ratio as a signal-to-noise ratio:

• Signal: How much do the group means differ from the overall mean? (Between-group variance)
• Noise: How much do individual scores vary around their own group mean? (Within-group variance)

If the models all have the same true mean, differences between group means are just sampling noise. The F-ratio will be near 1. If some model is genuinely better, the between-group variance inflates — F grows large.

The Dataset

Three model architectures evaluated on the same 5 random seeds:

Group means: ${\bar{X}}_{CNN}=0.785$, ${\bar{X}}_{ResNet}=0.836$, ${\bar{X}}_{Trans}=0.807$

Grand mean: ${\bar{X}}_{..}=(0.785+0.836+0.807)/3=0.809$

Note: this dataset uses independent observations per seed-model combination. For a paired (blocked) design with the same seeds, repeated measures ANOVA (post 16) is more powerful and appropriate.

ANOVA Hypotheses

$H_0:{\mu }_{CNN}={\mu }_{ResNet}={\mu }_{Trans}$ (all architectures perform equally)

$H_1$: At least one architecture differs

Computing SS Between Groups (Step by Step)

Between-group SS measures how far each group mean is from the grand mean:

$S{S}_B={\sum }_{i=1}^k{n}_i({\bar{X}}_{i.}-{\bar{X}}_{..}{)}^2$

$S{S}_B=5(0.785-0.809{)}^2+5(0.836-0.809{)}^2+5(0.807-0.809{)}^2$

$=5(-0.024{)}^2+5(0.027{)}^2+5(-0.002{)}^2$

$=5(0.000576)+5(0.000729)+5(0.000004)$

$=0.002880+0.003645+0.000020=0.006545$

Computing SS Within Groups (Step by Step)

Within-group SS measures variability of individual scores around their group mean:

$S{S}_W={\sum }_{i=1}^k{\sum }_{j=1}^{n_i}(X_{ij}-{\bar{X}}_{i.}{)}^2$

CNN: $(0.782-0.785{)}^2+(0.791-0.785{)}^2+(0.778-0.785{)}^2+(0.784-0.785{)}^2+(0.790-0.785{)}^2$ $=0.000009+0.000036+0.000049+0.000001+0.000025=0.000120$

ResNet: $(0.831-0.836{)}^2+(0.849-0.836{)}^2+(0.822-0.836{)}^2+(0.837-0.836{)}^2+(0.841-0.836{)}^2$ $=0.000025+0.000169+0.000196+0.000001+0.000025=0.000416$

Transformer: $(0.801-0.807{)}^2+(0.813-0.807{)}^2+(0.807-0.807{)}^2+(0.818-0.807{)}^2+(0.796-0.807{)}^2$ $=0.000036+0.000036+0.000000+0.000121+0.000121=0.000314$

$S{S}_W=0.000120+0.000416+0.000314=0.000850$

ANOVA Table

$M{S}_B=\frac{0.006545}{2}=0.003273$

$M{S}_W=\frac{0.000850}{12}=0.0000708$

$F=\frac{0.003273}{0.0000708}=46.22$

With $d{f}_1=2$, $d{f}_2=12$: the critical value $F_{0.05,2,12}=3.89$.

Since $46.22\gg 3.89$: reject $H_0$. The three architectures differ significantly in F1 performance.

Effect Size: Eta-Squared

${\eta }^2=\frac{S{S}_B}{S{S}_T}=\frac{0.006545}{0.007395}=0.885$

88.5% of the total variance in F1 scores is explained by architecture choice. This is a very large effect — architecture matters enormously here.

Post-Hoc Tests: Tukey HSD

ANOVA rejection tells you "at least one architecture differs" — not which ones. Tukey's Honestly Significant Difference test controls the family-wise error rate across all pairwise comparisons.

For each pair, compute the HSD threshold:

$HSD=q_{\alpha ,k,N-k}\times \sqrt{\frac{M{S}_W}{n}}$

With $q_{0.05,3,12}=3.77$ (studentized range statistic) and $M{S}_W=0.0000708$, $n=5$:

$HSD=3.77\times \sqrt{0.0000708/5}=3.77\times 0.00376=0.01418$

Pairwise differences:

• $|{\bar{X}}_{ResNet}-{\bar{X}}_{CNN}|=|0.836-0.785|=0.051>0.01418$ — significant
• $|{\bar{X}}_{Trans}-{\bar{X}}_{CNN}|=|0.807-0.785|=0.022>0.01418$ — significant
• $|{\bar{X}}_{ResNet}-{\bar{X}}_{Trans}|=|0.836-0.807|=0.029>0.01418$ — significant

All three architectures are significantly different from each other.

Non-Parametric Alternative: Kruskal-Wallis

When normality is violated, use the Kruskal-Wallis test — the non-parametric equivalent of one-way ANOVA. It ranks all observations jointly and tests whether ranks are distributed similarly across groups.

import numpy as np
from scipy import stats

cnn_scores    = np.array([0.782, 0.791, 0.778, 0.784, 0.790])
resnet_scores = np.array([0.831, 0.849, 0.822, 0.837, 0.841])
trans_scores  = np.array([0.801, 0.813, 0.807, 0.818, 0.796])

F_stat, p_value = stats.f_oneway(cnn_scores, resnet_scores, trans_scores)
print(f"ANOVA: F={F_stat:.4f}, p={p_value:.6f}")

grand_mean = np.concatenate([cnn_scores, resnet_scores, trans_scores]).mean()
ss_between = (5*(cnn_scores.mean() - grand_mean)**2 +
              5*(resnet_scores.mean() - grand_mean)**2 +
              5*(trans_scores.mean() - grand_mean)**2)
ss_total = np.sum((np.concatenate([cnn_scores, resnet_scores, trans_scores]) - grand_mean)**2)
eta_sq = ss_between / ss_total
print(f"Eta-squared: {eta_sq:.4f}")

# Tukey HSD post-hoc test
result = stats.tukey_hsd(cnn_scores, resnet_scores, trans_scores)
print(f"\nTukey HSD p-values:")
labels = ["CNN", "ResNet", "Transformer"]
for i in range(3):
    for j in range(i+1, 3):
        print(f"  {labels[i]} vs {labels[j]}: p={result.pvalue[i][j]:.4f}")

# Kruskal-Wallis (non-parametric alternative)
H_stat, p_kruskal = stats.kruskal(cnn_scores, resnet_scores, trans_scores)
print(f"\nKruskal-Wallis: H={H_stat:.4f}, p={p_kruskal:.6f}")

When ANOVA Does Not Apply

• Unequal variances: Use Welch's ANOVA (adjusts degrees of freedom)
• Non-normal residuals with small $n$: Use Kruskal-Wallis
• Same subjects across groups: Use repeated measures ANOVA (post 16)
• Very small samples: Check power carefully before interpreting results

Related Concepts

ANOVA extends the two-sample t-test (post 7) to $k>2$ groups by reframing the comparison in terms of variance. The F-statistic is the ratio of two chi-square variables divided by their degrees of freedom — connecting to post 12's chi-square framework. ANOVA's assumptions (normality, homoscedasticity, independence) are examined in detail in post 15. The different ANOVA designs (two-way, repeated measures, mixed) are covered in post 16. The multiple comparison problem that motivates ANOVA over repeated t-tests is the same problem addressed by Bonferroni and FDR corrections in post 9.

Honest Limitations

One-way ANOVA is a global test — rejecting $H_0$ only tells you that at least one group differs, not which one or by how much. Post-hoc tests like Tukey HSD answer the "which ones" question but use up some of the power. Eta-squared is slightly biased upward for small samples; omega-squared (${\omega }^2=(S{S}_B-(k-1)M{S}_W)/(S{S}_T+M{S}_W)$) is a less biased alternative. And with only 5 seeds per architecture, this ANOVA has limited power to detect small effects — the very large F here only works because the true architecture effects are enormous.

Test Your Understanding

1. You add a fourth architecture (Vision Transformer) with scores $[0.842,0.855,0.838,0.851,0.844]$. Compute $S{S}_B$ for all four groups with the new grand mean and explain how df changes.
2. The ANOVA F-ratio is described as a signal-to-noise ratio. What constitutes "signal" and what constitutes "noise" for the architecture comparison dataset? What would push F toward 1?
3. Tukey HSD found all three architectures significantly different. What additional information would a product team need beyond "significantly different" to decide which architecture to deploy?
4. The Kruskal-Wallis test gave $H=12.71$, $p=0.0017$. The parametric ANOVA gave $F=46.22$, $p<0.0001$. Why are these so different despite testing the same data?
5. Eta-squared for this dataset was 0.885. Without computing omega-squared, explain qualitatively whether you expect omega-squared to be higher or lower than 0.885 for these data, and why.