Cross-Validation

A single train/test split answers one question: how does this model perform on this particular partition of this particular dataset? Cross-validation answers the more useful question: how does this model perform on average, across all the ways you could have split the data? The difference matters most on small datasets, where a single split can be dominated by luck.

Anchor dataset: 6-sample house prices — small enough to trace every fold by hand.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import KFold, cross_val_score, LeaveOneOut

X = np.array([650, 850, 1100, 1400, 1600, 1900]).reshape(-1, 1)
y = np.array([180, 220, 280, 340, 370, 430])

The Problem with a Single Train/Test Split

With 6 samples and a 4/2 train/test split, the test set is just 2 data points. The MSE of those 2 predictions is your entire generalization estimate. If the test set happens to contain the two extreme samples (650 and 1900), the RMSE will be inflated. If both test samples fall in the middle (1100, 1400), the model looks better than it is.

This is high-variance estimation: the metric changes substantially depending on which 2 samples land in the test set. Cross-validation fixes this by using every sample as a test sample exactly once.

K-Fold Cross-Validation — Manual Trace

With $K=3$ and $n=6$: each fold trains on 4 samples and tests on 2. The 6 samples are partitioned into 3 non-overlapping groups.

Fold 1 trace — fit on samples 2–5:

Training: $x=[1100,1400,1600,1900]$, $y=[280,340,370,430]$

$\bar{x}=1500,\bar{y}=355$

$w_1=\frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x}{)}^2}=\frac{(-400)(-75)+(-100)(-15)+(100)(15)+(400)(75)}{160000+10000+10000+160000}=\frac{60000}{340000}\approx 0.1765$

$w_0=355-0.1765\times 1500=355-264.7=90.3$

Test predictions:

• $x=650$: $\hat{y}=90.3+0.1765\times 650=204.9$, $\epsilon =180-204.9=-24.9$, ${\epsilon }^2=620$
• $x=850$: $\hat{y}=90.3+0.1765\times 850=240.3$, $\epsilon =220-240.3=-20.3$, ${\epsilon }^2=412$

Fold 1 MSE $=(620+412)/2=516$

Fold 3 trace — fit on samples 0–3:

Training: $x=[650,850,1100,1400]$, $y=[180,220,280,340]$

$\bar{x}=1000,\bar{y}=255,w_1=0.2133,w_0=255-0.2133\times 1000=41.7$

Test predictions:

• $x=1600$: $\hat{y}=41.7+0.2133\times 1600=382.9$, $\epsilon =370-382.9=-12.9$, ${\epsilon }^2=166$
• $x=1900$: $\hat{y}=41.7+0.2133\times 1900=446.9$, $\epsilon =430-446.9=-16.9$, ${\epsilon }^2=286$

Fold 3 MSE $=(166+286)/2=226$

The fold MSEs vary considerably (Fold 1 > Fold 3) because 2-sample test sets are noisy. The CV estimate averages this variation out. These fold MSEs are also larger than the full training MSE of 22.2 — expected, because each fold trains on only 4 of 6 samples.

sklearn Implementation

kf = KFold(n_splits=3, shuffle=False)

mse_per_fold = []
for fold_idx, (train_idx, test_idx) in enumerate(kf.split(X)):
    X_tr, X_te = X[train_idx], X[test_idx]
    y_tr, y_te = y[train_idx], y[test_idx]
    
    model = LinearRegression()
    model.fit(X_tr, y_tr)
    y_pred = model.predict(X_te)
    fold_mse = ((y_te - y_pred) ** 2).mean()
    mse_per_fold.append(fold_mse)
    print(f"Fold {fold_idx+1}: Test MSE = {fold_mse:.2f}")

print(f"CV MSE = {np.mean(mse_per_fold):.2f} ± {np.std(mse_per_fold):.2f}")

The ±168 standard deviation on a mean of 284 shows how much fold-to-fold variance we have with only 2 test samples per fold. This is the signal to either increase $K$ or use LOOCV.

cross_val_score — Convenient Wrapper

scores = cross_val_score(
    LinearRegression(), X, y,
    cv=3,
    scoring='neg_mean_squared_error'
)
cv_mse = -scores.mean()
cv_std  = scores.std()
print(f"CV MSE: {cv_mse:.2f} ± {cv_std:.2f}")

sklearn uses neg_mean_squared_error because its CV infrastructure maximizes the score by default — negating MSE converts the minimization problem to a maximization one. Always negate when extracting the value.

Leave-One-Out Cross-Validation (LOOCV)

LOOCV is K-Fold with $K=n$: each fold trains on $n-1$ samples and tests on 1.

loo = LeaveOneOut()
scores = cross_val_score(
    LinearRegression(), X, y,
    cv=loo,
    scoring='neg_mean_squared_error'
)
print(f"LOOCV MSE per sample: {-scores.round(1)}")
print(f"LOOCV Mean MSE: {-scores.mean():.4f}")

LOOCV MSE (20.8) is close to full-data Train MSE (22.2) — expected for linear regression on a near-linear dataset. Note that samples 3 and 4 (sq_ft = 1100 and 1400) have higher individual LOO errors: those middle values are hardest to predict when only 5 other samples define the line.

K-Fold vs LOOCV Tradeoff

Nested Cross-Validation — Hyperparameter Tuning Without Leaking

A common mistake: use CV to select $\lambda$ and then report that same CV score as the model's performance estimate. The problem: you searched over $\lambda$ using all available data — the test folds influenced which $\lambda$ was chosen. The reported score is optimistically biased.

The correct structure is nested CV:

from sklearn.model_selection import GridSearchCV, KFold, cross_val_score
from sklearn.linear_model import Ridge

# Inner CV: selects best lambda within each outer training set
inner_cv = KFold(n_splits=3, shuffle=True, random_state=42)
# Outer CV: evaluates the final model on data not seen during lambda selection
outer_cv = KFold(n_splits=3, shuffle=True, random_state=0)

param_grid = {'alpha': [0.1, 1, 10, 100]}
gs = GridSearchCV(Ridge(), param_grid, cv=inner_cv, scoring='neg_mean_squared_error')

outer_scores = cross_val_score(gs, X, y, cv=outer_cv, scoring='neg_mean_squared_error')
print(f"Nested CV MSE: {-outer_scores.mean():.2f} ± {-outer_scores.std():.2f}")

In each outer fold:

1. The inner CV searches over alpha = [0.1, 1, 10, 100] using only the outer training set.
2. The best alpha is refitted on the outer training set.
3. Performance is evaluated on the outer test fold — data the inner CV never saw.

The outer CV score is an unbiased estimate of the model's performance after hyperparameter selection. It will typically be worse than a non-nested estimate because it's honest.

Choosing K in Practice

For time series data (TimeSeries Split in sklearn), shuffling is invalid — future data cannot be used to predict the past. Each fold must use only earlier time steps for training and later ones for testing.

CV Flavors Summary

Related Concepts and Honest Limitations

CV gives an estimate of generalization error — it doesn't guarantee it. If the entire dataset has a systematic bias (only expensive houses from one city), CV will accurately estimate how the model generalizes to that biased distribution, not to all house prices. CV improves the reliability of the estimate; it cannot fix data collection problems.

On our 6-sample anchor, the CV standard deviation of ±168 is larger than the mean of 284. This means the error bar spans from roughly 116 to 453 — not a useful estimate. The honest conclusion: 6 samples is too few for reliable CV. Cross-validation needs at least 50–100 samples to produce stable estimates; with fewer, LOOCV is the best available option but still has high variance.

Test Your Understanding

1. K-Fold with $K=n$ is identical to LOOCV. K-Fold with $K=1$ would train on zero samples. What is the minimum useful $K$, and what does the extreme $K=2$ represent?

2. The LOOCV MSE (20.83) is close to the full-data Train MSE (22.22) for linear regression. For a highly non-linear model like a degree-7 polynomial, would you expect LOOCV MSE to be close to or far from train MSE? Why?

3. In nested CV, the inner CV selects $\lambda =1$ for every outer fold. If the same $\lambda$ wins every time, is nested CV still necessary? What does this consistency tell you about the hyperparameter landscape?

4. You run 5-fold CV on a 1000-sample dataset and get MSE = 0.54 ± 0.12. You then run 10-fold CV and get MSE = 0.52 ± 0.08. The mean barely changed but variance dropped. Explain geometrically why more folds reduces variance of the CV estimate.

5. A colleague argues: "I'll use a 90/10 train/test split and just repeat it 10 times with different random seeds, then average the 10 test MSEs. Isn't that equivalent to 10-fold CV?" Is it? What's the key structural difference?