Cost Function in Linear Regression

You've computed two possible slopes — $w_1=0.10$ and $w_1=0.20$ — and both give different prediction errors. How do you compare them? You need a single number that tells you which set of weights is better. That number is the cost, and the way it changes as you adjust the weights defines the terrain the optimizer must cross.

This terrain is not just any landscape — for linear regression it's a perfect bowl. No local minima, no plateaus, no saddle points. Understanding why it's a bowl is what gives you confidence that every optimization method in this series — OLS, gradient descent — will converge to the same unique best answer.

From Loss to Cost

A loss function measures error for a single sample: $L({\hat{y}}_i,y_i)=(y_i-{\hat{y}}_i{)}^2$.

A cost function averages that loss over all $n$ training samples:

$J(w_0,w_1)=\frac{1}{n}{\sum }_{i=1}^n(y_i-w_0-w_1{x}_i{)}^2$

This is the Mean Squared Error (MSE). Some textbooks use $\frac{1}{2n}$ to cancel the factor of 2 from the derivative — this changes the scale but not the minimizer.

Why MSE and not MAE? MSE is differentiable everywhere. MAE has a non-differentiable kink at zero. Differentiability is what allows gradient descent and the OLS closed form to work.

Anchor dataset: $X=[650,850,1100,1400,1600,1900]$, $y=[180,220,280,340,370,430]$. True OLS solution: $w_0=53.33$, $w_1=0.20$.

MSE for Three Candidate Models

Hold $w_0=53.33$ fixed and vary $w_1$ to see how MSE changes:

$w_1=0.10$ (under-estimating the slope):

Predictions: $[118.3,138.3,163.3,193.3,213.3,243.3]$

Residuals: $[61.7,81.7,116.7,146.7,156.7,186.7]$

SSE ≈ 120,000. MSE ≈ 20,000.

$w_1=0.20$ (optimal slope):

Predictions: $[183.3,223.3,273.3,333.3,373.3,433.3]$

Residuals: $[-3.3,-3.3,6.7,6.7,-3.3,-3.3]$

SSE = 133.3. MSE = 22.2.

$w_1=0.30$ (over-estimating the slope):

Predictions: $[248.3,308.3,383.3,473.3,533.3,623.3]$

Residuals: $[-68.3,-88.3,-103.3,-133.3,-163.3,-193.3]$

SSE ≈ 120,000. MSE ≈ 20,000.

The minimum at $w_1=0.20$ gives MSE = 22.2. Deviating in either direction increases MSE rapidly — the parabola is steep.

The Full Loss Surface — 3D View (Both $w_0$ and $w_1$)

When you free both parameters, the cost function is a paraboloid — a bowl that curves upward in every direction from a single minimum at $(w_0=53.33,w_1=0.20)$.

This shape is guaranteed because MSE is a sum of squares — quadratic in $\mathbf{w}$. A quadratic has exactly one minimum for linear regression. That's what makes gradient descent safe here: no local minima to get trapped in.

Why MSE Is Convex — The Key Property

A function $f$ is convex if $f(\lambda a+(1-\lambda )b)\le \lambda f(a)+(1-\lambda )f(b)$ for all $a,b,\lambda \in [0,1]$.

MSE is quadratic in $\mathbf{w}$, so its Hessian is $H=\frac{2}{n}{X}^{\top }X$. If $X$ has full column rank (no redundant features), $H$ is positive definite — meaning the surface curves upward in every direction — and MSE is strictly convex with a unique global minimum.

Practical implication: gradient descent on MSE with linear regression will always converge to the optimal weights, regardless of starting point or step size (as long as the step size is small enough).

Cost Functions for Other Scenarios

Computing MAE and RMSE at the optimal weights:

Residuals: $[-3.3,-3.3,6.7,6.7,-3.3,-3.3]$

$\text{MAE}=\frac{3.3+3.3+6.7+6.7+3.3+3.3}{6}=\frac{26.6}{6}=4.43$

$\text{RMSE}=\sqrt{22.2}=4.71$

RMSE and MAE are close here (4.71 vs 4.43, ratio ≈ 1.06) because the residuals are uniform — only two distinct values: 3.3 and 6.7. When a model has a few large outlier-driven errors, this ratio grows significantly.

Plotting the Loss Curve

import numpy as np
import matplotlib.pyplot as plt

X = np.array([650, 850, 1100, 1400, 1600, 1900])
y = np.array([180, 220, 280, 340, 370, 430])
w0 = 53.33

w1_values = np.linspace(0.05, 0.35, 100)
mse_values = [np.mean((y - (w0 + w1 * X)) ** 2) for w1 in w1_values]

plt.plot(w1_values, mse_values)
plt.axvline(x=0.20, color='red', linestyle='--', label='Optimal w₁=0.20')
plt.xlabel('w₁ (slope)')
plt.ylabel('MSE')
plt.title('Loss Landscape — MSE vs w₁')
plt.legend()
plt.show()

# Output: parabola curve with minimum at w₁=0.20, MSE≈22.2
# Steep rise in both directions confirms unique global minimum

MSE Trace at the Optimal Weights

Related Concepts and Honest Limitations

The bowl shape is specific to linear regression with MSE. The moment you wrap the linear output in a sigmoid (logistic regression) or stack it in multiple layers (neural networks), the loss landscape gains saddle points — regions where the gradient is zero but the point is not a minimum. Gradient descent is no longer guaranteed to find the global optimum; it finds a good local one, which in practice is usually sufficient but theoretically weaker.

A common trap is assuming MSE is always the right cost because it's differentiable. MAE is not differentiable at zero, but that doesn't mean you can't minimize it — subgradient methods and coordinate descent handle the kink. The practical choice between them comes down to outlier tolerance. In this series we use MSE because it gives us the clean OLS closed form, but on real data with significant outliers, Huber loss splits the difference: quadratic for small errors, linear for large ones.

Test Your Understanding

1. For $w_1=0.10$ (holding $w_0=53.33$), compute the MSE manually by listing each residual and squaring it. Verify it's approximately 20,000.

2. The Hessian of MSE is $H=\frac{2}{n}{X}^{\top }X$. For the 1-feature anchor, compute $X^{\top }X$ (a 2×2 matrix with the bias column included) and verify it's positive definite.

3. Why does MSE penalize outliers more than MAE? Sketch two error scenarios — one with uniform small errors, one with one large error — and compare MSE vs MAE for each.

4. If you use $J=\frac{1}{2n}\sum {\epsilon }_i^2$ instead of $\frac{1}{n}\sum {\epsilon }_i^2$, does the optimal $w_0$ and $w_1$ change? Does the gradient change?

5. Huber loss transitions from quadratic to linear at $|\epsilon |=\delta$. What would the loss curve shape look like as a function of $w_1$? Would it still have a unique minimum?