Z-Test

Your team has been logging API response latency for six months. You know from historical data — millions of requests — that the population mean latency is ${\mu }_0=120$ ms with population standard deviation $\sigma =35$ ms. You deploy a new caching layer and measure latency on a sample of $n=64$ recent requests. The sample mean drops to $\bar{X}=108$ ms. Is that improvement real, or within normal random variation?

When you genuinely know $\sigma$ — not just estimate it — the Z-test is the right tool. In practice, knowing $\sigma$ is rare, but this scenario legitimately qualifies: six months of production data gives you a stable, reliable estimate of $\sigma$ that can be treated as known. Understanding Z-tests thoroughly also builds the intuition that carries over to every t-test you will ever run.

The Dataset

• Historical population: ${\mu }_0=120$ ms, $\sigma =35$ ms (known from production logs)
• Sample after caching: $\bar{X}=108$ ms, $n=64$ requests
• Observed improvement: 12 ms (10% reduction)

When Z-Tests Are Valid

The Z-test applies when either:

1. Population standard deviation $\sigma$ is truly known (from historical data at scale), or
2. Sample size is large enough ($n\ge 30$ typically) that the CLT makes $s\approx \sigma$ and the distinction between Z and t becomes minor

The honest distinction: When $\sigma$ is known exactly, Z is exact. When CLT justifies approximating $s\approx \sigma$ with large $n$, Z is an approximation — valid, but an approximation. The t-test is always the safer choice when $\sigma$ is estimated, because it accounts for the uncertainty in $s$.

The Z-statistic standardizes your sample mean: $Z=\frac{\bar{X}-{\mu }_0}{\sigma /\sqrt{n}}$

Under $H_0$, this follows a standard normal distribution $N(0,1)$.

Six-Step Procedure

Step 1: State hypotheses. We suspect latency decreased — but since we want to detect any change (caching could also introduce overhead in some edge cases):

$H_0:\mu =120$ ms (latency unchanged)

$H_1:\mu \ne 120$ ms (latency changed — two-tailed)

Significance level: $\alpha =0.05$

Step 2: Verify conditions. $\sigma =35$ ms known from historical data, $n=64$. CLT conditions satisfied.

Step 3: Calculate the test statistic.

$SE=\frac{\sigma }{\sqrt{n}}=\frac{35}{\sqrt{64}}=\frac{35}{8}=4.375\text{ ms}$

$Z=\frac{108-120}{4.375}=\frac{-12}{4.375}=-2.743$

Step 4: Find the critical value.

For $\alpha =0.05$ two-tailed: $Z_{critical}=\pm 1.96$

Step 5: Calculate the p-value.

$\text{p-value}=2\times P(Z<-2.743)=2\times 0.0030=0.0061$

Step 6: Decision.

$|-2.743|=2.743>1.96$: reject $H_0$. The latency improvement is statistically significant at $\alpha =0.05$.

Effect Size

Statistical significance does not tell you how meaningful the improvement is. Cohen's d for a one-sample test:

$d=\frac{\bar{X}-{\mu }_0}{\sigma }=\frac{108-120}{35}=\frac{-12}{35}=-0.343$

A Cohen's d of $|0.343|$ is a small-to-medium effect. The improvement is real but modest relative to the spread of latency values.

Power Analysis

Power is the probability of correctly rejecting $H_0$ when it is false. For a two-tailed Z-test with $n=64$, $\sigma =35$, $\alpha =0.05$, and true mean ${\mu }_1=108$ ms:

The non-centrality is $\delta /SE=12/4.375=2.743$.

$\text{Power}=P(Z<-1.96+2.743)+P(Z>1.96+2.743)$ $=P(Z<0.783)+P(Z>4.703)$ $\approx 0.783+0.000=0.783$

The test has 78% power for a 12 ms improvement — decent, but below the typical 80% target. To achieve 80% power for this effect size, you would need $n=68$ requests.

import numpy as np
from scipy import stats

mu_0 = 120   # historical mean latency (ms)
sigma = 35   # known population std (ms)
n = 64       # sample size
x_bar = 108  # sample mean after caching

se = sigma / np.sqrt(n)
z_stat = (x_bar - mu_0) / se
p_value = 2 * stats.norm.cdf(z_stat)  # z is negative, so cdf gives left tail

print(f"Standard error: {se:.3f} ms")
print(f"Z statistic: {z_stat:.4f}")
print(f"p-value: {p_value:.4f}")
print(f"Reject H0: {p_value < 0.05}")

# 95% confidence interval
z_crit = stats.norm.ppf(0.975)
ci_lower = x_bar - z_crit * se
ci_upper = x_bar + z_crit * se
print(f"95% CI: [{ci_lower:.2f}, {ci_upper:.2f}] ms")

# Power calculation
mu_1 = 108  # hypothesized true mean under H1
ncp = (mu_0 - mu_1) / se  # non-centrality parameter (positive)
power = stats.norm.cdf(-z_crit + ncp) + (1 - stats.norm.cdf(z_crit + ncp))
print(f"Power: {power:.4f}")

# Cohen's d
cohens_d = (x_bar - mu_0) / sigma
print(f"Cohen's d: {cohens_d:.4f}")

The Three Variants

One-sample Z-test: Compare a sample mean to a known population mean. Used above.

Two-sample Z-test: Compare means of two independent groups with known ${\sigma }_1,{\sigma }_2$. $Z=\frac{{\bar{X}}_1-{\bar{X}}_2}{\sqrt{{\sigma }_1^2/n_1+{\sigma }_2^2/n_2}}$

Z-test for proportions: Compare a sample proportion to a hypothesized value. $Z=\frac{\hat{p}-p_0}{\sqrt{p_0(1-p_0)/n}}$

Proportions tests are common in A/B testing (post 3 used this variant).

Critical Values Reference

When NOT to Use Z-Tests

When $\sigma$ is unknown and $n$ is small: Use t-tests. Using Z with an estimated standard deviation inflates your Type I error rate — your stated 5% false positive rate becomes 7-10% in practice.

For paired data: Use paired t-tests. Treating dependent observations as independent violates the independence assumption.

For non-normal data with small samples: The CLT does not apply. Consider non-parametric alternatives.

Relationship to Confidence Intervals

A $(1-\alpha )$ confidence interval for the mean: $\bar{X}\pm Z_{\alpha /2}\frac{\sigma }{\sqrt{n}}=108\pm 1.96\times 4.375=[99.4,116.6]\text{ ms}$

Since ${\mu }_0=120$ ms is not in $[99.4,116.6]$, we reject $H_0$ — consistent with the test result. The interval adds information: the true mean is plausibly anywhere from 99 to 117 ms, not necessarily exactly 108 ms.

Related Concepts

The Z-test builds on the CLT (post 1) to justify Normal sampling distributions for means. It is the purest form of the hypothesis testing procedure introduced in post 3. The t-test (posts 6 and 7) is a direct generalization that handles unknown $\sigma$ — the comparison is developed in post 8. Power analysis connects to Type I and Type II errors (post 9), where the tradeoff between $\alpha$ and $\beta$ is explored in full. The confidence interval for this example is developed in detail in post 11.

Honest Limitations

The Z-test's main limitation in practice is the rarity of truly known $\sigma$. Even production logs from millions of requests can have non-stationarities — latency distributions shift with load, software updates, and infrastructure changes. If $\sigma$ itself has changed since your historical baseline, the Z-test uses the wrong denominator. Always validate that your historical $\sigma$ is still representative before treating it as known.

Test Your Understanding

1. You change the cache implementation and now have $n=100$ samples with $\bar{X}=115$ ms. Using $\sigma =35$ ms from history, test $H_0:\mu =120$ at $\alpha =0.05$. What is the Z statistic and decision?
2. Calculate the power of the original test ($n=64$, $\sigma =35$, true mean 108 ms) for a one-tailed test at $\alpha =0.05$. Is it higher or lower than the two-tailed power of 78%?
3. A colleague argues that since the sample is large enough ($n=64$), you could use the sample standard deviation $s$ instead of the known $\sigma$ and still use Z-critical values. Is this correct? When exactly does this approximation become acceptable?
4. The 95% CI for latency after caching was $[99.4,116.6]$ ms. The operations team requires latency below 110 ms to meet SLA. Does the confidence interval guarantee the SLA is met?
5. You run the same Z-test on a one-tailed basis ($H_1:\mu <120$) because you only care about latency decreases. How does the critical value and p-value change? When is this one-tailed framing justified?