P-Value

The p-value might be the most misunderstood statistic in existence. It appears in every A/B test report, every model evaluation, and virtually every scientific paper — yet professional data scientists routinely interpret it wrong. The misinterpretation is not subtle: people believe the p-value measures the probability that their finding is real. It does not.

The p-value answers exactly one question: if the null hypothesis were true, how surprising would this data be? That is all it tells you.

The Dataset

Returning to the A/B test from post 3: version A served 5,000 users with CTR 3.20%, version B served 5,000 users with CTR 3.70%. The Z statistic was 1.938. We will use this concrete example throughout.

The Formal Definition

The p-value is the probability of obtaining test results at least as extreme as the results actually observed, under the assumption that the null hypothesis is correct:

$\text{p-value}=P(\text{observed or more extreme results}\mid H_0\text{ is true})$

For our A/B test with $Z_{obs}=1.938$ (two-tailed):

$\text{p-value}=2\times P(Z>1.938)=2\times 0.0263=0.0526$

This means: if version A and version B truly performed identically, there is a 5.26% chance of observing a CTR difference as large as 0.5pp or larger just from random sampling. The data is mildly surprising under $H_0$, but not decisively so.

What P-Values Are NOT

These misconceptions appear constantly in data science practice:

1. The p-value is NOT the probability that $H_0$ is true.

This would be $P(H_0\mid \text{data})$, which requires Bayesian inference. What we have is $P(\text{data}\mid H_0)$. By Bayes' theorem, $P(H_0\mid \text{data})\propto P(\text{data}\mid H_0)\times P(H_0)$. Even a small p-value can correspond to a high probability that $H_0$ is true, if the prior probability of an effect is low.

2. The p-value is NOT the probability that results are due to chance.

It tells you the probability of the data under the chance model, not whether chance produced the result.

3. The p-value is NOT a measure of effect size.

$p=0.001$ does not mean a larger CTR difference than $p=0.05$. With large $n$, even a 0.001 CTR difference produces tiny p-values. Effect size and p-value are related through sample size, not directly.

4. The p-value is NOT the probability of replication.

If a study produces $p=0.05$, repeating it gives $p<0.05$ only about 50% of the time, even if the effect is real. Replication probability depends on power, not the p-value.

Calculating the P-Value: Six-Step Walkthrough

For the A/B test (two-tailed, $\alpha =0.05$):

Step 1: Compute the pooled proportion under $H_0$: ${\hat{p}}_{pool}=(160+185)/10000=0.0345$

Step 2: Compute the standard error: $SE=\sqrt{0.0345\times 0.9655\times 2/5000}=0.00258$

Step 3: Compute the Z statistic: $Z=(0.037-0.032)/0.00258=1.938$

Step 4: Find the tail probability: $P(Z>1.938)=1-\Phi (1.938)=0.0263$

Step 5: Double for two-tailed: $\text{p-value}=2\times 0.0263=0.0526$

Step 6: Compare to $\alpha$: $0.0526>0.05$, fail to reject $H_0$.

The Rejection Region and P-Value Visualized

For this A/B test with $Z_{obs}=1.938$, the observed statistic sits just inside the non-rejection zone. The p-value is the total probability in both tails beyond the observed Z.

P-Value Distribution Under $H_0$ and $H_1$

Under $H_0$, p-values are uniformly distributed on $[0,1]$. This means exactly $\alpha$ of all p-values fall below $\alpha$ by chance — which is the definition of Type I error. Under $H_1$, p-values skew toward zero; the stronger the true effect and the larger the sample, the more concentrated they are near zero.

This uniform-under-$H_0$ property is useful: if you run many A/B tests and the p-value histogram is not uniform, something is wrong (p-hacking, data leakage, or optional stopping).

Power: What p = 0.0526 Also Tells You

The A/B test marginally failed significance. Does that mean version B is no better? Not necessarily. Power analysis (computed in post 3) showed only 49% power for a true 0.5pp difference. The test had a coin-flip chance of detecting the effect even if it were real. A p-value just above 0.05 with low power is weak evidence of no effect — it is mainly evidence of an underpowered experiment.

The Multiple Testing Problem

If you run 20 A/B tests across 20 different features simultaneously at $\alpha =0.05$, and all null hypotheses are true, you expect about 1 significant result purely by chance.

Solutions:

Bonferroni: Set ${\alpha }_{adjusted}=\alpha /m$ where $m$ is the number of tests. For 20 tests: ${\alpha }_{adjusted}=0.0025$. Conservative but simple.

False Discovery Rate (FDR): Control the expected proportion of false discoveries among significant results. More powerful than Bonferroni when running many tests.

Pre-registration: Specify hypotheses before analyzing data, preventing selective reporting.

P-Values and Effect Size: They Are Not the Same

Statistical significance is not practical significance. Consider two scenarios comparing our A/B test CTR difference:

The large experiment achieves significance for a 0.01% CTR difference. The current test barely misses significance for a 0.5% difference. Which finding matters more for your product? The 0.5% difference at $p=0.053$, clearly. Always report and consider effect sizes alongside p-values.

Python Code

import numpy as np
from scipy import stats

# A/B test: same data as post 3
clicks_a, n_a = 160, 5000
clicks_b, n_b = 185, 5000

p_a = clicks_a / n_a
p_b = clicks_b / n_b
p_pool = (clicks_a + clicks_b) / (n_a + n_b)

se = np.sqrt(p_pool * (1 - p_pool) * (1/n_a + 1/n_b))
z_obs = (p_b - p_a) / se

p_value = 2 * (1 - stats.norm.cdf(abs(z_obs)))
one_tail_area = 1 - stats.norm.cdf(abs(z_obs))

print(f"Z statistic: {z_obs:.4f}")
print(f"One-tail area: {one_tail_area:.4f}")
print(f"Two-tailed p-value: {p_value:.4f}")
print(f"Reject at alpha=0.05: {p_value < 0.05}")

# Demonstrate uniform distribution of p-values under H0
np.random.seed(42)
p_values_under_h0 = []
for _ in range(10000):
    a = np.random.binomial(5000, 0.035) / 5000
    b = np.random.binomial(5000, 0.035) / 5000
    p_hat = (5000*a + 5000*b) / 10000
    se_sim = np.sqrt(p_hat * (1-p_hat) * 2/5000)
    z = (b - a) / se_sim if se_sim > 0 else 0
    p_values_under_h0.append(2 * (1 - stats.norm.cdf(abs(z))))

below_alpha = np.mean(np.array(p_values_under_h0) < 0.05)
print(f"\nUnder H0, fraction of p-values below 0.05: {below_alpha:.4f} (should be ~0.05)")

The Replication Crisis Connection

The reproducibility problems in psychology, medicine, and data science trace directly to p-value misuse. Common issues:

P-hacking: Running multiple analyses until something significant appears, then reporting only that analysis as if it were pre-planned.

HARKing: Hypothesizing After Results are Known — presenting post-hoc hypotheses as if pre-specified.

Publication bias: Journals preferentially publish significant results, creating a literature that overestimates effect sizes.

Optional stopping: Checking results after each batch of users and stopping when $p<0.05$. This inflates Type I error to around 26% even with $\alpha =0.05$.

The American Statistical Association has released statements urging: do not use p-values as thresholds for truth, always consider context and study quality, and report effect sizes and confidence intervals alongside p-values.

Related Concepts

P-values connect back to the full hypothesis testing procedure (post 3) and the CLT (post 1) that justifies using the Normal distribution for test statistics. They connect forward to Type I and Type II errors (post 9), where the relationship between $\alpha$, $\beta$, and power is developed numerically. Confidence intervals (post 11) provide the same information in a more useful form: rather than a binary "reject or not," an interval shows the range of plausible effect sizes. For categorical outcomes, the chi-square test (post 12) uses the same p-value logic applied to a different test statistic.

Honest Limitations

P-values are designed for one purpose: quantifying discrepancy between data and a null hypothesis. They cannot tell you whether a finding will replicate, whether an effect is practically meaningful, or whether your model is correctly specified. A single p-value from a single experiment is weak evidence. Replication, pre-registration, effect size reporting, and confidence intervals together are stronger evidence.

Test Your Understanding

1. For the A/B test with $Z=1.938$, compute the one-tailed p-value for $H_1:p_B>p_A$. Is it significant at $\alpha =0.05$? Should you use this one-tailed test for your conclusion?
2. A researcher runs 100 A/B tests, all genuinely testing null effects, at $\alpha =0.05$. How many significant results do you expect? If they publish only those results, what is the published false discovery rate?
3. Two A/B tests compare different features: Test 1 has $p=0.04$ and a 0.02% CTR lift on $n=1,000,000$. Test 2 has $p=0.08$ and a 1.5% CTR lift on $n=200$. Which result is more actionable, and why?
4. Explain in one sentence why $P(\text{data}\mid H_0)\ne P(H_0\mid \text{data})$, and give a scenario from model evaluation where this distinction matters.
5. The p-value for a model comparison is $p=0.001$. A colleague says this means the probability of a false positive is 0.1%. Is this correct? What does the p-value actually mean in this context?