Z-Test vs t-Test

You are comparing two model versions. You run the experiment and face the first choice: Z-test or t-test? The short answer: use the t-test almost always. But understanding why — and exactly where the boundary lies — reveals something important about how uncertainty compounds in inference.

The Dataset

Two recommendation algorithm variants are compared on click-through rate (CTR). This is the same A/B test scenario used in posts 3 and 4, now viewed through the lens of which test is appropriate:

• Small pilot study: $n=12$ requests per arm, CTR_A = 8/12 = 0.667, CTR_B = 10/12 = 0.833
• Large production run: $n=5000$ requests per arm (post 3's data)

For the pilot: the team must decide between Z and t. For the production run, the choice barely matters.

What Actually Differs

Both tests answer the same question — is the observed difference consistent with $H_0$? — but they make different assumptions:

The distribution comparison shows why the choice matters at small $n$:

The Key Insight: Unknown Variance Compounds Uncertainty

When you use $s$ in place of $\sigma$, you are estimating a random variable — $s$ itself varies from sample to sample. The t-distribution absorbs this extra uncertainty through heavier tails. With small samples, $s$ can be a poor estimate of $\sigma$, so the tails are very heavy. As $n$ grows, $s$ converges to $\sigma$, and the t-distribution converges to the standard normal.

Pilot Study: Where the Choice Matters

For the pilot with $n=12$ per arm, suppose you treat CTR as approximately Normal (not ideal for binary data, but illustrative):

Sample means: ${\bar{X}}_A=0.667$, ${\bar{X}}_B=0.833$

Sample SDs: $s_A=0.236$, $s_B=0.183$ (pooled: $s_p=0.211$)

$SE=s_p\sqrt{1/n_A+1/n_B}=0.211\sqrt{2/12}=0.0863$

$t=(0.833-0.667)/0.0863=1.92$

Using t (correct for $df=22$): $t_{crit}=2.074$. Since $1.92<2.074$: fail to reject $H_0$.

Using Z (incorrect): $Z_{crit}=1.96$. Since $1.92<1.96$: still fail to reject $H_0$ here — but the margin is tighter and for slightly larger observed differences, Z would incorrectly reject.

The Error Inflation at Small n

The table below quantifies what happens if you use Z critical values when you should use t:

With $n=5$, using Z doubles your false positive rate. With $n=100$, the difference is negligible.

Power Analysis for Choosing Between Tests

For the production run ($n=5000$ per arm), both tests produce nearly identical results. For the pilot ($n=12$ per arm), the t-test has meaningfully less power due to wider critical values. If you need 80% power to detect a 0.1-unit CTR difference with $\sigma \approx 0.2$:

$n\approx \frac{(z_{\alpha /2}+z_{\beta }{)}^2\times 2{\sigma }^2}{{\delta }^2}=\frac{(1.96+0.842{)}^2\times 2\times 0.04}{0.01}=314\text{ per arm}$

Your pilot with 12 per arm has only about 12% power for this effect. The Z vs t question becomes moot if the study is severely underpowered.

Python Code

import numpy as np
from scipy import stats

np.random.seed(42)
n_simulations = 10000
n_small = 10
mu = 0.70  # true CTR
sigma = 0.20
alpha = 0.05

z_crit = stats.norm.ppf(0.975)
t_crit_small = stats.t.ppf(0.975, n_small - 1)

rejections_z = 0
rejections_t = 0

for _ in range(n_simulations):
    sample = np.random.normal(mu, sigma, n_small)
    sample_mean = np.mean(sample)
    sample_std = np.std(sample, ddof=1)
    se = sample_std / np.sqrt(n_small)

    z_stat_val = (sample_mean - mu) / se
    t_stat_val = (sample_mean - mu) / se  # same statistic, different critical value

    if abs(z_stat_val) > z_crit:
        rejections_z += 1
    if abs(t_stat_val) > t_crit_small:
        rejections_t += 1

print(f"Target Type I error: {alpha}")
print(f"Z-test Type I error: {rejections_z/n_simulations:.4f}  (inflated)")
print(f"t-test Type I error: {rejections_t/n_simulations:.4f}  (correct)")
print(f"\nZ critical value: {z_crit:.4f}")
print(f"t critical value (df={n_small-1}): {t_crit_small:.4f}")

Mathematical Relationship

As $n\to \infty$: $t_{n-1}\stackrel{d}{\to }N(0,1)$

The convergence is fast: by $n=30$, the difference in critical values is less than 5%. By $n=100$, it is less than 2%. This is why large-sample textbooks sometimes use Z throughout — but the t-test costs nothing extra computationally and is always correct.

Decision Rule

When in doubt, use the t-test. It gives the correct answer whether you have 5 observations or 5 million. The Z-test is a special case of the t-test at $df=\infty$.

Related Concepts

The Z vs t distinction builds on the t-distribution (post 6), which derives why replacing $\sigma$ with $s$ requires heavier tails. It resolves a question left open by the Z-test (post 5): what happens when $\sigma$ is not known? Post 7 shows the t-test in all three variants. The same reasoning behind Welch's adjusted degrees of freedom — that unequal variance reduces effective information — applies in ANOVA (posts 14–16) when comparing multiple groups. Power analysis (post 9) shows that this choice also affects how many samples you need for a given effect size.

Honest Limitations

Even the t-test has assumptions: independence of observations, and approximate normality (relaxed by CLT for large $n$). For proportions and counts, neither Z nor t is ideal for small samples — use exact binomial tests or Fisher's exact test. For severely non-Normal data at any sample size, consider bootstrap confidence intervals or non-parametric tests (Mann-Whitney U, Wilcoxon signed-rank) rather than either Z or t.

Test Your Understanding

1. For the pilot study ($n=12$ per arm), compute the exact Type I error probability if Z critical values are used with df = 22. Use the t-distribution's CDF to find $P(|t_{22}|>1.96)$.
2. A team has $n=50$ CTR observations per arm with $s_A=0.15$, $s_B=0.22$. They want to use a pooled t-test. What is the key assumption being made, and is it likely to hold given the observed standard deviations?
3. You know from a year of production logs that CTR has population standard deviation $\sigma =0.18$. You are testing a new arm with $n=20$ observations. Are you justified in using a Z-test, and why does the answer depend on more than just knowing $\sigma$?
4. Both Z and t give $|stat|>$ critical value for the production run ($n=5000$). A colleague says "so it doesn't matter which we use." Is this always true, or are there scenarios at large $n$ where the choice still matters?
5. Calculate how many more samples you need in the pilot (beyond $n=12$ per arm) to bring the pilot's power above 80% for a true 0.15-unit CTR difference, using $\sigma =0.20$ and $\alpha =0.05$.