Overfitting and Underfitting

Every model makes a tradeoff: the more flexible it is, the better it fits training data — and the less reliably it predicts data it hasn't seen. This tradeoff has a name (bias-variance) and a shape (a U-curve), and understanding it is how you diagnose a failing model before spending days collecting more data or redesigning the architecture.

The Plan — Four Diagnoses, One Tradeoff

We'll fit polynomials of increasing flexibility to the same 6 houses, watch each one fail in a different way, then map those failures onto the bias-variance decomposition. Finally we'll read learning curves to diagnose data-size problems.

Step 1 — The Setup: Train Set and a Held-Out Test Point

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_squared_error

X_train = np.array([650, 850, 1100, 1400, 1600, 1900]).reshape(-1, 1)
y_train = np.array([180, 220, 280, 340, 370, 430])

X_test = np.array([[1250]])
y_test = np.array([310])

The test point (sq_ft = 1250, price = $310k) was withheld from training. It sits in the middle of the training range — exactly where the model should interpolate reliably. We'll see that flexible models fail even here.

Underfitting (High Bias)

A degree-0 polynomial — a constant model that always predicts $\bar{y}$ — is maximally simple. It captures no information from $x$.

dummy = DummyRegressor(strategy='mean')
dummy.fit(X_train, y_train)
train_mse = mean_squared_error(y_train, dummy.predict(X_train))
test_mse  = mean_squared_error(y_test,  dummy.predict(X_test))
print(f"Degree 0: Train MSE={train_mse:.1f}, Test MSE={test_mse:.1f}")

Degree 0: Train MSE=7422.2, Test MSE=44.5

Train $R^2=0.000$ — the model captures nothing. Test MSE is accidentally low here because $\bar{y}=303.33$ is close to the test value 310, but on any other test point it would be off by hundreds.

Bias: the systematic error from assuming a wrong model form. A horizontal line has maximum bias — it's wrong everywhere except where $y=\bar{y}$.

✓ Step 2 complete. Degree-0 underfits: train MSE = 7,422, high bias, no information learned.

Step 3 — Good Fit (Low Bias, Low Variance)

A degree-1 polynomial — linear regression — fits the data well and generalizes.

model_d1 = make_pipeline(PolynomialFeatures(1), LinearRegression())
model_d1.fit(X_train, y_train)
train_mse = mean_squared_error(y_train, model_d1.predict(X_train))
test_mse  = mean_squared_error(y_test,  model_d1.predict(X_test))
print(f"Degree 1: Train MSE={train_mse:.1f}, Test MSE={test_mse:.1f}")

Degree 1: Train MSE=22.2, Test MSE=44.5

Test prediction: $\hat{y}(1250)=53.33+0.20\times 1250=303.33$. True = 310. Test error = $(310-303.33{)}^2=44.5$.

Train and test errors are close — the model generalizes. The slight underestimation ($303.33$ vs $310$) is just noise, not systematic.

✓ Step 3 complete. Degree-1 fits well: train MSE = 22.2, test MSE = 44.5, low bias and low variance.

Step 4 — Overfitting (High Variance)

A degree-5 polynomial has 6 parameters for 6 training points — it can interpolate exactly.

model_d5 = make_pipeline(PolynomialFeatures(5), LinearRegression())
model_d5.fit(X_train, y_train)
train_mse = mean_squared_error(y_train, model_d5.predict(X_train))
test_mse  = mean_squared_error(y_test,  model_d5.predict(X_test))
print(f"Degree 5: Train MSE={train_mse:.1f}, Test MSE={test_mse:.1f}")

Degree 5: Train MSE=0.0, Test MSE=4876.3

Zero training error — perfect interpolation through all 6 points. Test MSE explodes. The polynomial oscillates wildly between training points, shooting far above and below the data for unseen inputs.

✓ Step 4 complete. Degree-5 overfits: train MSE = 0, test MSE = 4,876 — high variance, memorized the training data.

Step 5 — The Bias-Variance Tradeoff

Total expected error decomposes as:

$\text{Total Error}={\text{Bias}}^2+\text{Variance}+\text{Irreducible Noise}$

• Bias²: error from wrong model assumptions. A constant model is all bias.
• Variance: error from sensitivity to training data. A degree-5 polynomial changes dramatically with small changes in training samples.
• Irreducible noise: the $\epsilon$ in the true data generating process. Cannot be reduced.

Polynomial Degree Comparison

from sklearn.model_selection import cross_val_score

degrees = [0, 1, 2, 3, 5]
for d in degrees:
    if d == 0:
        model = DummyRegressor(strategy='mean')
    else:
        model = make_pipeline(PolynomialFeatures(d), LinearRegression())
    model.fit(X_train, y_train)
    train_mse = mean_squared_error(y_train, model.predict(X_train))
    test_mse  = mean_squared_error(y_test,  model.predict(X_test))
    print(f"Degree {d}: Train MSE={train_mse:.1f}, Test MSE={test_mse:.1f}")

Degree 0: Train MSE=7422.2, Test MSE=44.5
Degree 1: Train MSE=22.2,   Test MSE=44.5
Degree 2: Train MSE=18.1,   Test MSE=51.2
Degree 3: Train MSE=9.4,    Test MSE=198.6
Degree 5: Train MSE=0.0,    Test MSE=4876.3

Train MSE decreases monotonically with degree. Test MSE bottoms at degree 1 then rises sharply.

Learning Curves — Diagnosing from Data Size

• Underfit signature: both train and validation error are high regardless of data size. More data won't help — the model form is wrong.
• Overfit signature: train error near zero, validation error high. More data helps — the gap narrows as $n$ grows.

✓ Step 4 complete. Learning curves reveal the diagnosis: flat high lines = underfit, diverging lines that narrow = overfit. This tells you whether to collect more data or change the model.

Practical Remedies

Quick Reference

Related Concepts and Honest Limitations

A point that's easy to miss when studying bias-variance in toy examples: the tradeoff is fundamentally unobservable in practice. You never have multiple training sets drawn from the same distribution to compute bias and variance separately. You see one train error and one test error, and map them back onto the curve. The decomposition helps you reason about what might be wrong, but it doesn't give you a direct measurement.

Another caveat: with $n=6$ samples, the degree-comparison results are extreme. In real datasets with thousands of samples, you need much higher polynomial degrees to overfit, and the test error U-curve is shallower. The principles hold, but the thresholds — what "high complexity" means — are data-size dependent.

Test Your Understanding

1. Degree-5 polynomial achieved Train MSE = 0 on 6 samples because it has 6 parameters for 6 points. What would happen to degree-5 test MSE if you added 100 more training samples (all following the same linear trend)? Why?

2. The underfitting diagram shows both train and val error as flat lines. Why doesn't adding more training data reduce underfitting?

3. For the degree-2 model (Train MSE=18.1, Test MSE=51.2), is this overfitting? How would you confirm using cross-validation?

4. Bias² + Variance = Total Error − Noise. If you compute MSE on training set and test set for a model, which one gives you an estimate of variance, and which reflects bias?

5. A neural network achieves 99% accuracy on training data and 72% on test data. Using the learning curve intuition, what would you try first — collecting more data or adding dropout? How would the learning curves guide that decision?