Ridge, Lasso, and ElasticNet Regression

OLS finds the weights that minimize SSE — with no constraint on how large those weights can be. When features are nearly collinear, $X^{\top }X$ becomes near-singular and the OLS solution explodes: tiny changes in the training data produce huge swings in the coefficients. Regularization fixes this by adding a penalty term to the cost function that discourages large weights. The result is a biased but far more stable estimator — and for Lasso, a sparse one that zeros out irrelevant features entirely.

The Problem Regularization Solves

Our anchor dataset has corr(sq_ft, bedrooms) ≈ 0.997. In post 11, the OLS solution gave $w_1=0.175$ and $w_2=9.61$. These are technically optimal — but fragile. On a dataset where sqft and bedrooms are this correlated, the model could assign $w_1=+500,w_2=-490$ and achieve nearly the same SSE, because any extra value added by sqft can be offset by subtracting from bedrooms. OLS picks one of infinitely many near-equivalent solutions.

Add a small perturbation to one training sample and OLS might pick a completely different near-equivalent solution. High coefficient magnitude = high variance = generalization failure.

Regularization adds a penalty for coefficient size to the cost function. The model now balances two objectives: fit the training data (minimize MSE) and keep weights small (minimize penalty). The tradeoff is controlled by $\lambda$.

Ridge Regression (L2 Regularization)

Cost function:

$J(\mathbf{w})=\underset{\text{MSE}}{\underbrace{\frac{1}{n}\sum _{i=1}^n(y_i-{\hat{y}}_i{)}^2}}+\lambda \underset{\text{L2 penalty}}{\underbrace{\sum _{j=1}^p{w}_j^2}}$

The bias term $w_0$ is typically excluded from the penalty — we want to shrink feature weights, not the intercept.

Closed-form solution:

${\mathbf{w}}^{\ast }=(X^{\top }X+\lambda I{)}^{-1}{X}^{\top }\mathbf{y}$

Adding $\lambda I$ to $X^{\top }X$ is the key. Even if $X^{\top }X$ is singular (near-collinear features), $X^{\top }X+\lambda I$ is always invertible for any $\lambda >0$. As $\lambda \to 0$, Ridge converges to OLS. As $\lambda \to \infty$, all feature weights converge to 0 (only the intercept remains).

Manual trace on the 2-feature anchor ($\lambda =10$):

From post 11, $X^{\top }X$ has diagonal $[6,10985000,62]$. Adding $\lambda I$ makes it $[16,10985010,72]$. At this scale, $\lambda =10$ is negligible relative to the diagonal magnitude — $w_1$ barely changes from 0.175. This is why Ridge (and Lasso) must always be applied to standardized features: on the original scale, features with large magnitudes dominate and $\lambda$ has inconsistent effects.

Ridge on Scaled California Housing

import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score

data = fetch_california_housing()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

for name, model in [
    ('OLS',          LinearRegression()),
    ('Ridge(λ=0.1)', Ridge(alpha=0.1)),
    ('Ridge(λ=1)',   Ridge(alpha=1)),
    ('Ridge(λ=10)',  Ridge(alpha=10)),
    ('Ridge(λ=100)', Ridge(alpha=100)),
]:
    pipe = make_pipeline(StandardScaler(), model)
    pipe.fit(X_train, y_train)
    y_pred = pipe.predict(X_test)
    print(f"{name:20s}: R²={r2_score(y_test, y_pred):.4f}, RMSE={np.sqrt(mean_squared_error(y_test, y_pred)):.4f}")

$\lambda =1$ is the sweet spot — a tiny improvement over OLS because it stabilizes the near-collinear AveRooms/AveBedrms pair. At $\lambda =100$, performance degrades: over-regularized, weights are shrunk too much.

Ridge Coefficient Shrinkage

from sklearn.linear_model import Ridge

scaler = StandardScaler()
X_sc = scaler.fit_transform(X_train)

lambdas = [0, 0.1, 1, 10, 100, 1000]
print(f"{'Lambda':>8} | {'MedInc':>8} | {'HouseAge':>8} | {'AveRooms':>8} | {'Latitude':>8}")
for lam in lambdas:
    m = LinearRegression() if lam == 0 else Ridge(alpha=lam)
    m.fit(X_sc, y_train)
    c = m.coef_
    print(f"{lam:>8} | {c[0]:>8.4f} | {c[1]:>8.4f} | {c[2]:>8.4f} | {c[6]:>8.4f}")

All coefficients shrink toward 0 as $\lambda$ increases — but none reach exactly 0. Ridge is not a feature selection method: it keeps every feature in the model, just with smaller weights.

Lasso Regression (L1 Regularization)

Cost function:

$J(\mathbf{w})=\frac{1}{n}{\sum }_{i=1}^n(y_i-{\hat{y}}_i{)}^2+\lambda {\sum }_{j=1}^p|w_j|$

The absolute value penalty $\lambda \Vert \mathbf{w}{\Vert }_1$ is not differentiable at $w_j=0$, so there is no closed-form solution. Lasso is solved via coordinate descent: cycle through features one at a time and minimize the cost with respect to each $w_j$ while holding all others fixed. The solution for each step has an analytic form involving the soft-threshold operator.

from sklearn.linear_model import Lasso

lambdas = [0.01, 0.05, 0.1, 0.5, 1.0]
print(f"{'Lambda':>8} | {'MedInc':>8} | {'HouseAge':>8} | {'AveRooms':>8} | {'Latitude':>8} | Nonzero")
for lam in lambdas:
    pipe = make_pipeline(StandardScaler(), Lasso(alpha=lam, max_iter=10000))
    pipe.fit(X_train, y_train)
    c = pipe.named_steps['lasso'].coef_
    nz = np.sum(c != 0)
    print(f"{lam:>8} | {c[0]:>8.4f} | {c[1]:>8.4f} | {c[2]:>8.4f} | {c[6]:>8.4f} | {nz}/8")

At $\lambda =0.1$, HouseAge and AveRooms are zeroed out — Lasso has automatically removed two features. At $\lambda =0.5$, only MedInc and one other survive. This is automatic feature selection built into the penalty.

Each line kinks to zero at a different $\lambda$ and stays there. The horizontal segments on the zero line are exactly zero — the feature has been removed.

Why L1 Creates Sparsity but L2 Doesn't — Geometric Explanation

Think of regularization as a constrained optimization. Minimizing MSE subject to a budget on weights. The budget constraint is:

• L2 (Ridge): $\sum w_j^2\le B$ — a circle in 2D parameter space (smooth, no corners)
• L1 (Lasso): $\sum |w_j|\le B$ — a diamond in 2D parameter space (corners at the axes)

The unconstrained OLS solution is the center of the MSE contour ellipses. As $\lambda$ increases (budget $B$ decreases), the ellipses expand outward until they first touch the constraint region.

For L1 (left): the first touch of the expanding ellipse with the diamond is almost always at a corner — where one coordinate is exactly zero. For L2 (right): the smooth circle has no corners, so the first touch is always off-corner — both coordinates are nonzero, just small.

ElasticNet — Combining L1 and L2

When you want sparsity (L1) but also want to handle groups of correlated features gracefully (L2), use ElasticNet:

$J(\mathbf{w})=\frac{1}{n}\sum (y_i-{\hat{y}}_i{)}^2+{\lambda }_1\Vert \mathbf{w}{\Vert }_1+{\lambda }_2\Vert \mathbf{w}{\Vert }_2^2$

In sklearn: alpha = λ₁ + λ₂ and l1_ratio = λ₁/(λ₁+λ₂). Setting l1_ratio=1 gives Lasso; l1_ratio=0 gives Ridge.

from sklearn.linear_model import ElasticNet
from sklearn.model_selection import GridSearchCV

param_grid = {
    'elasticnet__alpha':    [0.01, 0.1, 0.5, 1.0],
    'elasticnet__l1_ratio': [0.1, 0.5, 0.9]
}
pipe = make_pipeline(StandardScaler(), ElasticNet(max_iter=10000))
gs = GridSearchCV(pipe, param_grid, cv=5, scoring='neg_mean_squared_error')
gs.fit(X_train, y_train)

print(f"Best params: {gs.best_params_}")
print(f"Best CV MSE: {-gs.best_score_:.4f}")

Lasso (when two features are collinear) tends to arbitrarily pick one and zero the other. ElasticNet with l1_ratio=0.5 keeps both but shrinks them proportionally — better interpretability when correlated features are both meaningful.

Choosing λ — Built-in Cross-Validation

from sklearn.linear_model import RidgeCV

ridge_cv = make_pipeline(
    StandardScaler(),
    RidgeCV(alphas=[0.01, 0.1, 1, 10, 100])
)
ridge_cv.fit(X_train, y_train)
print(f"RidgeCV best alpha: {ridge_cv.named_steps['ridgecv'].alpha_:.4f}")

RidgeCV and LassoCV perform efficient leave-one-out (Ridge) or coordinate-descent-based (Lasso) cross-validation internally. They are faster than GridSearchCV for scalar hyperparameter search and should be the default when only $\lambda$ needs tuning.

Ridge vs Lasso vs ElasticNet

Related Concepts and Honest Limitations

The "instability of OLS under collinearity" claim requires a concrete demonstration to be convincing — run OLS on the same dataset with one training point perturbed and observe the coefficient swings. With corr ≈ 0.997, OLS weights can move by hundreds while predictions barely change. Ridge prevents this by restricting the feasible region for weights.

The limitation: the optimal $\lambda$ is dataset-dependent and must always be chosen by cross-validation. There is no principled way to choose $\lambda$ from the training data alone — any heuristic risks either underfitting or overfitting the penalty. Always run at minimum a 5-fold CV before fixing $\lambda$.

Lasso's automatic feature selection is a strength and a risk. When two features are collinear, Lasso picks one arbitrarily — which one it picks can change with a different random seed or dataset split. If both features are scientifically meaningful, use ElasticNet rather than letting Lasso discard one.

Test Your Understanding

1. The Ridge closed form is $(X^{\top }X+\lambda I{)}^{-1}{X}^{\top }\mathbf{y}$. As $\lambda \to \infty$, what does ${\mathbf{w}}^{\ast }$ converge to? Derive it from the formula.

2. Ridge on the California Housing dataset shows minimal improvement over OLS ($R^2$ changes by < 0.001 at $\lambda =1$). Why might Ridge still be the preferred model even when the test metric barely changes?

3. Lasso zeroed out HouseAge at $\lambda =0.1$. Does this mean HouseAge has no predictive value for house prices? What would you do to check?

4. At $\lambda =1.0$, Lasso leaves only 1/8 features nonzero. If you use this sparse model's predictions as input to another model (stacking), does the L1 sparsity benefit transfer?

5. The geometric argument shows L1 hits corners while L2 doesn't. In 3D parameter space ($w_1,w_2,w_3$), what does the L1 constraint region look like, and where are its "corners"? Does the same sparsity argument hold?