Simple Linear Regression

A real estate agent asks you: "I have six recent sales in this neighborhood. If a 1,000 sq ft house comes on the market, what's it worth?" You could take the average — $303k — but you'd ignore the obvious pattern: bigger houses sell for more. You need a formula that turns square footage into a price. That's simple linear regression: one input, one output, find the best line.

The derivation is short enough to do by hand, and doing it by hand is how you internalize why the slope formula involves covariance and variance — not just what the formula is. This is not the same as a lookup table where you check the closest past sale; a linear model gives you a prediction for any square footage, even ones you've never seen.

Anchor dataset:

import numpy as np
from sklearn.linear_model import LinearRegression

X = np.array([650, 850, 1100, 1400, 1600, 1900]).reshape(-1, 1)
y = np.array([180, 220, 280, 340, 370, 430])

The Plan — Three Steps to Simple Linear Regression

We'll work through three steps: first measure how bad a naive guess is, then derive the formulas for the optimal slope and intercept, then compute them by hand on our anchor data and verify with code.

Step 1 — The Loss: Sum of Squared Errors

The residual for sample $i$ is ${\epsilon }_i=y_i-{\hat{y}}_i=y_i-(w_0+w_1{x}_i)$.

The loss is the sum of squared errors:

$\text{SSE}={\sum }_{i=1}^n{\epsilon }_i^2={\sum }_{i=1}^n(y_i-w_0-w_1{x}_i{)}^2$

Why squared? It penalizes large errors more than small ones, treats over- and under-prediction symmetrically, and is differentiable everywhere — properties that allow a closed-form solution.

As a baseline, try the worst possible model: $w_0=w_1=0$ (always predict zero):

${\text{SSE}}_{\text{naive}}=180^2+220^2+280^2+340^2+370^2+430^2$ $=32400+48400+78400+115600+136900+184900=596600$

Our goal is to find $w_0$ and $w_1$ that drive SSE far below this.

✓ Step 1 complete. SSE for the naive all-zeros model is 596,600. Every step from here drives this number down.

Step 2 — Deriving the OLS Formulas

We start by setting the partial derivatives of SSE with respect to $w_0$ and $w_1$ to zero — this finds the bottom of the bowl-shaped loss surface:

$\frac{\partial \text{SSE}}{\partial w_0}=-2\sum (y_i-w_0-w_1{x}_i)=0⟹n{w}_0+w_1\sum x_i=\sum y_i$

$\frac{\partial \text{SSE}}{\partial w_1}=-2\sum x_i(y_i-w_0-w_1{x}_i)=0⟹w_0\sum x_i+w_1\sum x_i^2=\sum x_i{y}_i$

Solving this $2\times 2$ system yields the OLS normal equations:

$w_1=\frac{\sum x_i{y}_i-n\bar{x}\bar{y}}{\sum x_i^2-n{\bar{x}}^2}=\frac{\text{Cov}(x,y)}{\text{Var}(x)}$

$w_0=\bar{y}-w_1\bar{x}$

The slope is the covariance between $x$ and $y$ divided by the variance of $x$ — how much $y$ moves per unit movement in $x$, adjusted for $x$'s own spread.

✓ Step 2 complete. We have the formulas: $w_1=\text{Cov}(x,y)/\text{Var}(x)$ and $w_0=\bar{y}-w_1\bar{x}$.

Step 3 — Computing OLS by Hand on the Anchor

Step 1: Summary statistics

$\bar{x}=\frac{650+850+1100+1400+1600+1900}{6}=\frac{7500}{6}=1250$

$\bar{y}=\frac{180+220+280+340+370+430}{6}=\frac{1820}{6}=303.33$

Step 2: Per-sample products

Step 3: Compute weights

$w_1=\frac{222000}{1110000}=0.2000$

$w_0=303.33-0.2000\times 1250=303.33-250=53.33$

Final model: $\hat{y}=53.33+0.20\times \text{sq\_ft}$

✓ Step 3 complete. $w_1=0.2000$ and $w_0=53.33$. Each additional square foot adds $200 to predicted price.

Step 4 — Predictions and Residuals

SSE dropped from 596600 (naive model) to 133.33 — a 4500× reduction. No other linear model can achieve lower SSE on this data; OLS is provably optimal among all unbiased linear estimators.

✓ Step 4 complete. SSE at the optimal weights is 133.33, down from 596,600 at $w_0=w_1=0$.

Step 5 — Code Verification

model = LinearRegression()
model.fit(X, y)

print(f"w₀ (intercept): {model.intercept_:.2f}")
print(f"w₁ (slope):     {model.coef_[0]:.4f}")

w₀ (intercept): 53.33
w₁ (slope):     0.2000

The sklearn default uses ordinary least squares with no regularization — it should match our hand derivation exactly. Verify by predicting a new house:

new_house = np.array([[1000]])
print(f"Predicted price for 1000 sq_ft: ${model.predict(new_house)[0]:.1f}k")

Predicted price for 1000 sq_ft: $253.3k

Manual check: $53.33+0.20\times 1000=253.3\text{k}$ ✓

✓ Step 5 complete. sklearn confirms $w_0=53.33$, $w_1=0.2000$, prediction for 1,000 sq ft is $253.3k.

Assumptions

Four conditions are required for OLS to behave well:

1. Linearity — the true relationship is $E[y|x]=w_0+w_{1}x$.
2. Independence — residuals are independent across samples.
3. Homoscedasticity — the variance of residuals is constant (doesn't grow with $x$).
4. Normality — residuals follow a normal distribution (needed for inference, not prediction).

Violation check: plot residuals vs fitted values. Random scatter around zero confirms the assumptions. A fan shape means heteroscedasticity. A curved pattern means the linearity assumption is wrong.

Key Formulas

Related Concepts and Honest Limitations

The OLS closed form is unique to linear regression with MSE — it works because the loss is quadratic in the weights, which gives a single global minimum you can solve for directly. For logistic regression or neural networks, the loss has no such algebraic shortcut, which is why gradient descent (post 7) exists. Multiple linear regression (post 8) extends the same formulas to $p$ features using the matrix form ${\mathbf{w}}^{\ast }=(X^{\top }X{)}^{-1}{X}^{\top }\mathbf{y}$.

A common mistake here is assuming linearity is a small constraint. It's not — if the true relationship between sq_ft and price curves (prices that plateau at large houses), the best-fit line will have systematic residual patterns regardless of how carefully you tune the weights. You'll see it immediately in a residual-vs-fitted plot: the residuals won't scatter randomly around zero; they'll curve. That's the signal to add polynomial features or switch model class, not to keep tweaking what you have.

Test Your Understanding

1. For our 6-sample anchor, verify that the sum of residuals $\sum {\epsilon }_i=0$. Is this always true for OLS? Why?

2. If you doubled all $y$ values (prices in $2k increments instead of $k), how would $w_0$ and $w_1$ change? Use the OLS formulas to derive this, not trial and error.

3. The SSE dropped from 596600 to 133.33 with OLS weights. Could you find weights that give SSE = 0? If not, why not? If yes, what would that mean about the data?

4. What is the residual for a house with sq_ft = 1250? Is the model over- or under-predicting?

5. A colleague argues that maximizing $R^2$ is the same as minimizing SSE. Are they correct? Under what conditions would these objectives give different results?