Overfitting and Underfitting

Every model makes a tradeoff: the more flexible it is, the better it fits training data — and the less reliably it predicts data it hasn't seen. This tradeoff has a name (bias-variance) and a shape (a U-curve), and understanding it is how you diagnose a failing model before spending days collecting more data or redesigning the architecture.

The Setup: Train Set and a Held-Out Test Point

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_squared_error

X_train = np.array([650, 850, 1100, 1400, 1600, 1900]).reshape(-1, 1)
y_train = np.array([180, 220, 280, 340, 370, 430])

X_test = np.array([[1250]])
y_test = np.array([310])

The test point (sq_ft = 1250, price = $310k) was withheld from training. It sits in the middle of the training range — exactly where the model should interpolate reliably. We'll see that flexible models fail even here.

Underfitting (High Bias)

A degree-0 polynomial — a constant model that always predicts $\bar{y}$ — is maximally simple. It captures no information from $x$.

dummy = DummyRegressor(strategy='mean')
dummy.fit(X_train, y_train)
train_mse = mean_squared_error(y_train, dummy.predict(X_train))
test_mse  = mean_squared_error(y_test,  dummy.predict(X_test))
print(f"Degree 0: Train MSE={train_mse:.1f}, Test MSE={test_mse:.1f}")

Train $R^2=0.000$ — the model captures nothing. Test MSE is accidentally low here because $\bar{y}=303.33$ is close to the test value 310, but on any other test point it would be off by hundreds.

Bias: the systematic error from assuming a wrong model form. A horizontal line has maximum bias — it's wrong everywhere except where $y=\bar{y}$.

Good Fit (Low Bias, Low Variance)

A degree-1 polynomial — linear regression — fits the data well and generalizes.

model_d1 = make_pipeline(PolynomialFeatures(1), LinearRegression())
model_d1.fit(X_train, y_train)
train_mse = mean_squared_error(y_train, model_d1.predict(X_train))
test_mse  = mean_squared_error(y_test,  model_d1.predict(X_test))
print(f"Degree 1: Train MSE={train_mse:.1f}, Test MSE={test_mse:.1f}")

Test prediction: $\hat{y}(1250)=53.33+0.20\times 1250=303.33$. True = 310. Test error = $(310-303.33{)}^2=44.5$.

Train and test errors are close — the model generalizes. The slight underestimation ($303.33$ vs $310$) is just noise, not systematic.

Overfitting (High Variance)

A degree-5 polynomial has 6 parameters for 6 training points — it can interpolate exactly.

model_d5 = make_pipeline(PolynomialFeatures(5), LinearRegression())
model_d5.fit(X_train, y_train)
train_mse = mean_squared_error(y_train, model_d5.predict(X_train))
test_mse  = mean_squared_error(y_test,  model_d5.predict(X_test))
print(f"Degree 5: Train MSE={train_mse:.1f}, Test MSE={test_mse:.1f}")

Zero training error — perfect interpolation through all 6 points. Test MSE explodes. The polynomial oscillates wildly between training points, shooting far above and below the data for unseen inputs.

The Bias-Variance Tradeoff

Total expected error decomposes as:

$\text{Total Error}={\text{Bias}}^2+\text{Variance}+\text{Irreducible Noise}$

• Bias²: error from wrong model assumptions. A constant model is all bias.
• Variance: error from sensitivity to training data. A degree-5 polynomial changes dramatically with small changes in training samples.
• Irreducible noise: the $\epsilon$ in the true data generating process. Cannot be reduced.

Polynomial Degree Comparison

from sklearn.model_selection import cross_val_score

degrees = [0, 1, 2, 3, 5]
for d in degrees:
    if d == 0:
        model = DummyRegressor(strategy='mean')
    else:
        model = make_pipeline(PolynomialFeatures(d), LinearRegression())
    model.fit(X_train, y_train)
    train_mse = mean_squared_error(y_train, model.predict(X_train))
    test_mse  = mean_squared_error(y_test,  model.predict(X_test))
    print(f"Degree {d}: Train MSE={train_mse:.1f}, Test MSE={test_mse:.1f}")

Train MSE decreases monotonically with degree. Test MSE bottoms at degree 1 then rises sharply.

Learning Curves — Diagnosing from Data Size

• Underfit signature: both train and validation error are high regardless of data size. More data won't help — the model form is wrong.
• Overfit signature: train error near zero, validation error high. More data helps — the gap narrows as $n$ grows.

Practical Remedies

Quick Reference

Related Concepts and Honest Limitations

The bias-variance tradeoff is the fundamental reason regularization exists. Ridge and Lasso (post 14) explicitly shrink model complexity — they trade a small increase in bias for a large reduction in variance, which improves test error when the model is in the overfit regime.

One limitation of the degree-comparison experiment above: with $n=6$ samples, the results are extreme. In real datasets with thousands of samples, you need much higher polynomial degrees to overfit, and the test error U-curve is shallower. The principles hold, but the thresholds are data-size dependent.

Test Your Understanding

1. Degree-5 polynomial achieved Train MSE = 0 on 6 samples because it has 6 parameters for 6 points. What would happen to degree-5 test MSE if you added 100 more training samples (all following the same linear trend)? Why?

2. The underfitting diagram shows both train and val error as flat lines. Why doesn't adding more training data reduce underfitting?

3. For the degree-2 model (Train MSE=18.1, Test MSE=51.2), is this overfitting? How would you confirm using cross-validation?

4. Bias² + Variance = Total Error − Noise. If you compute MSE on training set and test set for a model, which one gives you an estimate of variance, and which reflects bias?

5. A neural network achieves 99% accuracy on training data and 72% on test data. Using the learning curve intuition, what would you try first — collecting more data or adding dropout? How would the learning curves guide that decision?