Hypothesis Testing

You ran an A/B test: version A of your recommendation algorithm served 5,000 users and produced a mean click-through rate of 3.2%, while version B served another 5,000 users and produced 3.7%. Version B looks better. But is that 0.5-percentage-point difference real, or is it just random fluctuation you would expect even if both algorithms performed identically?

This is the question hypothesis testing exists to answer. It forces you to be precise about what "real" means by asking: if there were truly no difference, how surprising would this data be? You do not prove things directly — you rule out the "nothing is happening" explanation indirectly.

The Two-Algorithm Dataset

Throughout this post, the dataset is:

• Version A: $n_A=5000$, mean CTR ${\bar{p}}_A=0.032$, observed clicks $=160$
• Version B: $n_B=5000$, mean CTR ${\bar{p}}_B=0.037$, observed clicks $=185$
• Observed difference: ${\hat{p}}_B-{\hat{p}}_A=0.005$

The Setup

Every hypothesis test starts with two competing claims:

The Null Hypothesis ($H_0$) is the status quo: "nothing special is happening." For the A/B test: $H_0:p_A=p_B$ — both algorithms perform identically.

The Alternative Hypothesis ($H_1$) is what you suspect is true: $H_1:p_A\ne p_B$ — the algorithms differ in CTR.

You collect data, calculate a test statistic, and ask: if $H_0$ were true, how surprising would this data be? If it is very surprising, you reject $H_0$.

One-Tailed vs Two-Tailed: Choose Before Seeing Data

This choice must be made before you look at results.

Two-tailed ($H_1:p_A\ne p_B$): You care about any difference — version B could be better or worse. Use this when you have no prior reason to expect a specific direction. It is the conservative, default choice for A/B tests because you do not know in advance which direction the effect will go.

One-tailed ($H_1:p_B>p_A$): You care only about version B being better. Use this only when (1) the opposite direction is impossible or irrelevant, and (2) you specified this before collecting data. Using one-tailed tests after you see the direction of the effect is a form of p-hacking.

For this A/B test, use two-tailed: you want to know if the algorithms differ, not just if B is better.

The Six-Step Procedure

Step 1: State hypotheses and choose tail direction.

$H_0:p_B-p_A=0$

$H_1:p_B-p_A\ne 0$ (two-tailed)

Significance level: $\alpha =0.05$

Step 2: Choose the appropriate test.

Both groups have large $n$ and binary outcomes. Use a two-proportion z-test. The pooled proportion under $H_0$ is: ${\hat{p}}_{pool}=\frac{{\text{clicks}}_A+{\text{clicks}}_B}{n_A+n_B}=\frac{160+185}{10000}=0.0345$

Step 3: Calculate the test statistic.

$SE=\sqrt{{\hat{p}}_{pool}(1-{\hat{p}}_{pool})\left(\frac{1}{n_A}+\frac{1}{n_B}\right)}=\sqrt{0.0345\times 0.9655\times \frac{2}{5000}}=0.00258$

$Z=\frac{{\hat{p}}_B-{\hat{p}}_A}{SE}=\frac{0.005}{0.00258}=1.938$

Step 4: Find the critical value and rejection region.

For $\alpha =0.05$ two-tailed: $Z_{critical}=\pm 1.96$.

Reject $H_0$ if $|Z|>1.96$.

Step 5: Calculate the p-value.

$\text{p-value}=2\times P(Z>1.938)=2\times (1-\Phi (1.938))=2\times 0.0263=0.0526$

Step 6: Decision and interpretation.

$|1.938|<1.96$ and $p=0.0526>0.05$: fail to reject $H_0$. The observed CTR difference is not statistically significant at $\alpha =0.05$. The data is consistent with random variation.

Rejection Region and p-Value SVGs

The rejection region is the set of Z values where you would reject $H_0$. The observed statistic sits just inside the non-rejection zone.

The p-value is the probability of observing Z at least as extreme as 1.938 under $H_0$.

Power: Can This Test Detect a Real Effect?

Power is the probability of correctly rejecting $H_0$ when a real difference exists. For the A/B test with $n=5000$ per group, $\alpha =0.05$, and true difference $\delta =0.005$:

$\text{Power}=P\left(|Z|>1.96\mid \delta =0.005\right)$

With $SE=0.00258$, the effect corresponds to $\delta /SE=0.005/0.00258=1.938$ standard errors. The non-centrality parameter is 1.938.

Power $=P(Z>1.96-1.938)+P(Z<-1.96-1.938)=P(Z>0.022)+P(Z<-3.898)$

$=(1-0.509)+0.0000\approx 0.49$

The test has only 49% power for this effect size — essentially a coin flip for detecting a real 0.5pp difference. To achieve 80% power, you would need roughly $n=15,000$ per group.

Python Code

import numpy as np
from scipy import stats

# A/B test data
clicks_a, n_a = 160, 5000
clicks_b, n_b = 185, 5000

p_a = clicks_a / n_a
p_b = clicks_b / n_b
p_pool = (clicks_a + clicks_b) / (n_a + n_b)

se = np.sqrt(p_pool * (1 - p_pool) * (1/n_a + 1/n_b))
z_stat = (p_b - p_a) / se
p_value = 2 * (1 - stats.norm.cdf(abs(z_stat)))

print(f"CTR A: {p_a:.4f}, CTR B: {p_b:.4f}")
print(f"Pooled proportion: {p_pool:.4f}")
print(f"Standard error: {se:.5f}")
print(f"Z statistic: {z_stat:.4f}")
print(f"p-value: {p_value:.4f}")
print(f"Reject H0: {p_value < 0.05}")

Common Mistakes

Testing after seeing data: Never define your hypotheses or choose one-tailed vs two-tailed after looking at results. This invalidates the inference.

Equating statistical significance with practical significance: A 0.005 CTR difference might be economically meaningful at scale even if $p=0.06$. Calculate effect size and business impact separately.

Confusing "fail to reject" with "accept $H_0$": Not rejecting $H_0$ means the data is consistent with no difference — it does not mean no difference exists.

Ignoring power: The A/B test above has only 49% power. An underpowered study is ethically questionable — you are spending resources on a study unlikely to detect real effects.

Multiple testing without correction: Running 20 A/B tests simultaneously at $\alpha =0.05$ gives roughly 1 false positive even if no algorithm actually differs.

Related Concepts

Hypothesis testing builds on the CLT (post 1) to justify using Normal or t distributions for test statistics. It uses estimation concepts (post 2) for computing standard errors. The p-value mechanics are expanded in post 4, and the specific test statistics for means and proportions are developed in posts 5 through 8. Type I and Type II errors (post 9) formalize the tradeoffs introduced here with power analysis. Confidence intervals (post 11) are the complement: everything this test rejects at level $\alpha$ is outside the $(1-\alpha )$ confidence interval.

Honest Limitations

Hypothesis testing answers one specific question: is this data consistent with the null hypothesis? It says nothing about why an effect exists, whether it will replicate, or whether it matters. An A/B test that narrowly fails to reject $H_0$ might still be worth acting on if the effect size is large enough and the cost of inaction is high. Statistical significance is a tool, not a decision-making algorithm.

Test Your Understanding

1. In the A/B test above, if you increase the sample to $n=10000$ per group with the same observed CTR rates, would the result change? Calculate the new Z statistic.
2. A team argues they should use a one-tailed test because "we expect version B to be better." What information would you need to know before accepting this choice of test?
3. The A/B test had 49% power for a true 0.5pp difference. What does this mean in plain language for a product manager who must decide whether to run the experiment?
4. You run 10 A/B tests simultaneously at $\alpha =0.05$. Two come back significant. How many would you expect to be false positives if none of the algorithms actually differ?
5. After seeing $p=0.0526$, a colleague rounds to "basically 5%" and declares the result significant. What is wrong with this reasoning?