Type I and Type II Errors

Your A/B test declared p=0.03 and you shipped a new model. Two months later a replication found no effect — you generated a false positive. Or: your test found p=0.12 and you shelved the model. A competitor built the same thing and gained market share — you missed a real effect. Both mistakes have costs. Every experiment navigates the tradeoff between them.

The Anchor

A/B test: new ranking model vs. control
σ = 0.08         # CTR standard deviation (from historical data)
δ_true = 0.005   # true effect: 0.5pp CTR improvement (if real)
α = 0.05         # significance level
n = 1,000        # users per arm

The Error Matrix

Before any formulas: four possible outcomes depending on ground truth × decision.

In the anchor context:

• True positive (power): real 0.5pp improvement detected → ship the model
• False positive (Type I): no real improvement, test says yes → ship a useless model
• True negative: no improvement, test says no → correct, don't ship
• False negative (Type II): real 0.5pp improvement missed → model sits on shelf, competitor ships it

Type I Error: False Positive Rate α

Definition: reject H₀ when it is actually true. You conclude there is an effect when there is none.

Rate: you set α directly. With α=0.05, if the null is truly true and you repeat the experiment 100 times, you incorrectly reject ~5 times. This is the cost you accept for the ability to detect real effects.

Why α=0.05 is a convention, not a law:

Relationship to critical value: α = P(|Z| > z* | H₀ true). For α=0.05 one-tailed: z* = 1.645.

Type II Error: False Negative Rate β

Definition: fail to reject H₀ when it is actually false. A real effect exists but the test misses it.

Rate: β is NOT set directly. It is determined by four factors:

1. n — larger sample → smaller β (more sensitive test)
2. δ — larger effect → smaller β (easier to detect)
3. σ — smaller variability → smaller β (cleaner signal)
4. α — larger α → smaller β (but more false positives)

Computing β for the anchor:

Under H₁, the test statistic Z has a non-central normal distribution with non-centrality parameter:

ncp = δ / SE = δ / (σ × √(2/n)) = 0.005 / (0.08 × √(2/1000)) = 0.005 / 0.003578 = 1.397

β = P(Z < z* | Z ~ N(ncp, 1)) = P(Z < 1.645 − 1.397) = P(Z < 0.248) = 0.598

Power = 1 − β = 0.402

With n=1,000 per arm, the test has only 40% power (one-tailed). It would miss the real 0.5pp improvement 60% of the time.

Statistical Power

Power = 1 − β = P(reject H₀ | H₁ true) = P(Z > z* | Z ~ N(ncp, 1))

Power formula (two-tailed): Power = Φ(−z* + ncp) + Φ(−z* − ncp)

For the anchor: Power = Φ(1.397 − 1.960) + Φ(−1.397 − 1.960) = Φ(−0.563) + Φ(−3.357) = 0.287 + 0.000 = 0.287 (two-tailed, n=1000)

Power vs n table:

α–β Trade-off at Fixed n

Reducing α (stricter threshold) increases β (more missed effects) — for fixed n. The only way to reduce both simultaneously is to increase n.

Sample Size Calculation

Goal: n per arm to achieve target power at given δ, σ, α.

Formula: n = (z_α/2 + z_β)² × 2σ² / δ²

Where z_β = Φ⁻¹(1−β) (quantile for the target power).

Substitution for 80% power (z_β=0.842), α=0.05 (z_α/2=1.960):

n = (1.960 + 0.842)² × 2 × 0.08² / 0.005² = (2.802)² × 2 × 0.0064 / 0.000025 = 7.851 × 0.0128 / 0.000025 = 4,021 per arm

Sample size table for different target powers:

Statistical vs Practical Significance

Large-n inflation: n=500,000 per arm with δ=0.0001 (0.01pp). ncp=0.625 → Power≈7% — this tiny effect is hard to detect even with half a million users. But if a well-powered study finds p<0.05 for δ=0.0001, a 0.01pp CTR improvement has no practical value in most product contexts.

The rule: always report effect size alongside p-value. A significant result with negligible effect should not drive deployment decisions.

Reference Tables

Error types:

Power factors:

Code and Output

import numpy as np
from scipy import stats

sigma = 0.08
delta = 0.005
alpha = 0.05
n = 1000

SE = sigma * np.sqrt(2 / n)
ncp = delta / SE
z_crit = stats.norm.ppf(1 - alpha/2)

power = stats.norm.cdf(-z_crit + ncp) + stats.norm.cdf(-z_crit - ncp)
beta = 1 - power

print(f"SE = {SE:.6f}")
print(f"z* = {z_crit:.3f},  ncp = δ/SE = {ncp:.4f}")
print(f"Power = {power:.4f},  β = {beta:.4f}")

print("\nα-β tradeoff (n=1000 per arm, δ=0.005, σ=0.08):")
for a in [0.01, 0.05, 0.10, 0.20]:
    z_a = stats.norm.ppf(1 - a/2)
    pwr = stats.norm.cdf(-z_a + ncp) + stats.norm.cdf(-z_a - ncp)
    print(f"  α={a:.2f}: z*={z_a:.3f}, power={pwr:.3f}, β={1-pwr:.3f}")

z_beta = stats.norm.ppf(0.80)
n_required = (z_crit + z_beta)**2 * 2 * sigma**2 / delta**2
print(f"\nRequired n per arm (power=0.80, α=0.05): {int(np.ceil(n_required))}")

print("\nSample size table:")
for target_power, label in [(0.70, "70%"), (0.80, "80%"), (0.90, "90%"), (0.95, "95%")]:
    z_b = stats.norm.ppf(target_power)
    n_req = (z_crit + z_b)**2 * 2 * sigma**2 / delta**2
    print(f"  Power={label}: n ≥ {int(np.ceil(n_req))}")

print("\nPower at different n (δ=0.005, σ=0.08, α=0.05):")
for n_test in [500, 1000, 2000, 4021, 6000]:
    se_t = sigma * np.sqrt(2 / n_test)
    ncp_t = delta / se_t
    pwr_t = stats.norm.cdf(-z_crit + ncp_t) + stats.norm.cdf(-z_crit - ncp_t)
    print(f"  n={n_test:>5}: ncp={ncp_t:.3f}, power={pwr_t:.3f}")

SE = 0.003578
z* = 1.960,  ncp = δ/SE = 1.3971
Power = 0.2869,  β = 0.7131

α-β tradeoff (n=1000 per arm, δ=0.005, σ=0.08):
  α=0.01: z*=2.576, power=0.152, β=0.848
  α=0.05: z*=1.960, power=0.287, β=0.713
  α=0.10: z*=1.645, power=0.402, β=0.598
  α=0.20: z*=0.842, power=0.589, β=0.411

Required n per arm (power=0.80, α=0.05): 4021

Sample size table:
  Power=70%: n ≥ 2457
  Power=80%: n ≥ 4021
  Power=90%: n ≥ 6093
  Power=95%: n ≥ 8061

Power at different n (δ=0.005, σ=0.08, α=0.05):
  n=  500: ncp=0.988, power=0.166
  n= 1000: ncp=1.397, power=0.287
  n= 2000: ncp=1.975, power=0.506
  n= 4021: ncp=2.801, power=0.800
  n= 6000: ncp=3.419, power=0.927

Test Your Understanding

1. The experiment with n=1,000 per arm has power=28.7%. After running the experiment, you get p=0.12 and fail to reject H₀. A stakeholder says "the test proved the model doesn't work." Write a two-sentence correction that accurately states what the result means, referencing the power calculation.

2. You want to reduce both Type I (α) and Type II (β) errors simultaneously. Your manager says "just use α=0.01 instead of 0.05 — that'll give us fewer false positives." Show numerically (using the anchor parameters) why this does NOT reduce both errors simultaneously at fixed n.

3. Two teams propose experiments: Team A wants to detect δ=0.005 with 90% power. Team B wants to detect δ=0.010 with 90% power. Both use σ=0.08 and α=0.05. Without computing exact numbers first, explain the relationship between required n for Team A vs Team B. Then compute both.

4. A published study reports p=0.04 with n=200 per group. A replication uses n=200 per group and finds p=0.18. A journalist says "the replication disproved the original finding." Using power analysis, explain why the "failed" replication is consistent with the original finding being real.

5. Your team runs 10 independent A/B tests simultaneously, each with α=0.05. What is the probability that at least one is a false positive, assuming all null hypotheses are true? Apply Bonferroni correction. If you apply Bonferroni, what is the new threshold for each individual test?