Skewness and Kurtosis

All three have the same mean and variance

Kurtosis ≠ Peakedness

The common description — "kurtosis measures how peaked a distribution is" — is wrong. Kurtosis measures tail weight. A leptokurtic distribution can have a lower peak than the normal distribution while still having heavier tails and higher kurtosis. The excess probability mass moves from the shoulders of the distribution into the tails; the peak height is a byproduct, not the cause. Describing kurtosis as peakedness leads to misinterpretation — you should watch the tails, not the top.

When Kurtosis Matters for ML

• Heavy tails (leptokurtic) during training: gradient updates from outlier batches can be explosive. Loss distributions during training instability are often leptokurtic. Gradient clipping addresses this.
• t-distribution: leptokurtic with excess kurtosis = 6 / (df − 4) for df > 4. As df → ∞, it converges to the normal (excess kurtosis → 0). Small-sample t-tests rely on this.
• Normality tests: Shapiro-Wilk and Jarque-Bera both check skewness and kurtosis jointly to detect departures from normality.

Jarque-Bera Normality Test

Both skewness and kurtosis are combined into a single normality test:

Under H₀ (data is normally distributed), skewness = 0 and excess kurtosis = 0, so JB ≈ 0. JB follows a chi-square distribution with 2 degrees of freedom asymptotically. A large JB → small p-value → reject normality.

Applied to both anchors:

from scipy import stats

accuracy   = [0.82, 0.79, 0.91, 0.85, 0.78, 0.88]
latency_ms = [120, 135, 142, 148, 155, 189, 312, 890]

jb_acc, p_acc = stats.jarque_bera(accuracy)
jb_lat, p_lat = stats.jarque_bera(latency_ms)

print(f"accuracy:   JB={jb_acc:.4f}, p={p_acc:.4f}")
print(f"latency_ms: JB={jb_lat:.4f}, p={p_lat:.4f}")

# skewness and excess kurtosis separately
print(f"\naccuracy   skewness={stats.skew(accuracy):.4f}, "
      f"kurtosis(excess)={stats.kurtosis(accuracy):.4f}")
print(f"latency_ms skewness={stats.skew(latency_ms):.4f}, "
      f"kurtosis(excess)={stats.kurtosis(latency_ms):.4f}")

Interpret:

• accuracy: JB=0.63, p=0.73 — do not reject normality. Skewness (0.28) is well within the symmetric range; the negative excess kurtosis (−1.48) reflects fewer tail events than a normal, which makes sense for six bounded CV scores.
• latency_ms: JB=23.56, p≈0 — reject normality strongly. Skewness=2.56 (highly right-skewed), excess kurtosis=6.72 (very heavy tails driven by the 890 ms spike). Apply a log transform or use nonparametric methods before any normality-assuming analysis.

Note: scipy's stats.kurtosis() returns excess kurtosis (Fisher's definition, subtracting 3). Pandas .kurt() also returns excess kurtosis by default.

Computing Skewness for the Latency Anchor

For completeness, here is the latency skewness breakdown:

The 890 ms entry alone contributes 13.963 / 13.440 ≈ 104% of the total skewness. Remove it and the distribution becomes approximately symmetric.

Practical Workflow

When you receive a new dataset, follow this sequence before choosing any statistical method:

The workflow: plot first (skewness is a confirmation, not a substitute for visual inspection) → quantify asymmetry → quantify tail weight → run a formal normality test → choose the right method.

Related Concepts and Honest Limitations

Related: The moments table connects directly to the moment-generating function used in probability theory. Skewness is central to understanding the central limit theorem — it tells you how many samples you need before the sample mean distribution is approximately normal (more skewness → more samples needed). Kurtosis connects to value-at-risk in finance and gradient stability in deep learning.

Limitations:

• With small samples (n < 30), skewness and kurtosis estimates have high variance. The Jarque-Bera test is asymptotic and unreliable for small n — prefer Shapiro-Wilk when n < 50.
• Skewness and kurtosis describe the marginal distribution of one variable. They say nothing about multivariate structure, dependencies, or conditional distributions.
• A skewness of 0 does not guarantee symmetry — a distribution can be asymmetric but have cubed deviations that cancel. Similarly, zero kurtosis does not mean the distribution is normal.

Test Your Understanding

1. You have model latency data with x̄ = 200 ms and median = 140 ms. Without computing skewness, what sign do you expect it to have, and why?

2. You remove the 890 ms outlier from the latency data, leaving seven values. How would you expect the skewness and excess kurtosis to change? Would the Jarque-Bera test still reject normality?

3. Two distributions have γ₁ = 0.0 (zero skewness). Does this guarantee they are identical in shape? Construct a counterexample or explain why not.

4. A colleague says: "This loss distribution has very high kurtosis, so it has a very sharp peak." What is wrong with this statement, and what does high kurtosis actually tell you about the loss distribution?

5. You are training a model and notice that the gradient distribution has excess kurtosis of 8.5. What does this suggest about the training dynamics, and what practical step would you take to address it?