Simple Linear Regression

Simple linear regression is the most constrained version of the problem: one input, one output, find the best line. The derivation is short enough to do by hand, and doing it by hand is how you internalize why the slope formula involves covariance and variance — not just what the formula is.

The Model

$\hat{y}=w_0+w_{1}x$

$w_0$ is the intercept, $w_1$ is the slope. "Simple" means one predictor. For every additional square foot, the predicted price increases by $w_1$ thousand dollars.

Anchor dataset:

import numpy as np
from sklearn.linear_model import LinearRegression

X = np.array([650, 850, 1100, 1400, 1600, 1900]).reshape(-1, 1)
y = np.array([180, 220, 280, 340, 370, 430])

The Loss: Sum of Squared Errors

The residual for sample $i$ is ${\epsilon }_i=y_i-{\hat{y}}_i=y_i-(w_0+w_1{x}_i)$.

The loss is the sum of squared errors:

$\text{SSE}={\sum }_{i=1}^n{\epsilon }_i^2={\sum }_{i=1}^n(y_i-w_0-w_1{x}_i{)}^2$

Why squared? It penalizes large errors more than small ones, treats over- and under-prediction symmetrically, and is differentiable everywhere — properties that allow a closed-form solution.

As a baseline, try the worst possible model: $w_0=w_1=0$ (always predict zero):

${\text{SSE}}_{\text{naive}}=180^2+220^2+280^2+340^2+370^2+430^2$ $=32400+48400+78400+115600+136900+184900=596600$

Our goal is to find $w_0$ and $w_1$ that drive SSE far below this.

Deriving the OLS Formulas

Set the partial derivatives to zero:

\frac{\partial \text{SSE}}{\partial w_0} = -2 \sum (y_i - w_0 - w_1 x_i) = 0 \implies n w_0 + w_1 \sum x_i = \sum y_i \tag{1}

\frac{\partial \text{SSE}}{\partial w_1} = -2 \sum x_i(y_i - w_0 - w_1 x_i) = 0 \implies w_0 \sum x_i + w_1 \sum x_i^2 = \sum x_i y_i \tag{2}

Solving this $2\times 2$ system yields the OLS normal equations:

$w_1=\frac{\sum x_i{y}_i-n\bar{x}\bar{y}}{\sum x_i^2-n{\bar{x}}^2}=\frac{\text{Cov}(x,y)}{\text{Var}(x)}$

$w_0=\bar{y}-w_1\bar{x}$

The slope is the covariance between $x$ and $y$ divided by the variance of $x$ — how much $y$ moves per unit movement in $x$, adjusted for $x$'s own spread.

Computing OLS by Hand on the Anchor

Step 1: Summary statistics

$\bar{x}=\frac{650+850+1100+1400+1600+1900}{6}=\frac{7500}{6}=1250$

$\bar{y}=\frac{180+220+280+340+370+430}{6}=\frac{1820}{6}=303.33$

Step 2: Per-sample products

Step 3: Compute weights

$w_1=\frac{222000}{1110000}=0.2000$

$w_0=303.33-0.2000\times 1250=303.33-250=53.33$

Final model: $\hat{y}=53.33+0.20\times \text{sq\_ft}$

Predictions and Residuals

SSE dropped from 596600 (naive model) to 133.33 — a 4500× reduction. No other linear model can achieve lower SSE on this data; OLS is provably optimal among all unbiased linear estimators.

Code Implementation

model = LinearRegression()
model.fit(X, y)

print(f"w₀ (intercept): {model.intercept_:.2f}")
print(f"w₁ (slope):     {model.coef_[0]:.4f}")

new_house = np.array([[1000]])
print(f"Predicted price for 1000 sq_ft: ${model.predict(new_house)[0]:.1f}k")

Manual check: 53.33 + 0.20 \times 1000 = \253.3k$ ✓

Assumptions

Four conditions are required for OLS to behave well:

1. Linearity — the true relationship is $E[y|x]=w_0+w_{1}x$.
2. Independence — residuals are independent across samples.
3. Homoscedasticity — the variance of residuals is constant (doesn't grow with $x$).
4. Normality — residuals follow a normal distribution (needed for inference, not prediction).

Violation check: plot residuals vs fitted values. Random scatter around zero confirms the assumptions. A fan shape means heteroscedasticity. A curved pattern means the linearity assumption is wrong.

Key Formulas

Related Concepts and Honest Limitations

The OLS formula is a closed-form solution unique to linear regression on MSE. For logistic regression or neural networks, the loss is no longer quadratic in the weights, so no closed form exists — gradient descent is required. Multiple linear regression extends this to $p$ features using the matrix form ${\mathbf{w}}^{\ast }=(X^{\top }X{)}^{-1}{X}^{\top }\mathbf{y}$, which reduces to the formulas here when $p=1$.

The limitation of simple linear regression is its linearity. If the true relationship between sq_ft and price is non-linear — plateauing at large houses, for example — the best-fit line will have systematic residual patterns (curvature in the residual-vs-fitted plot). That's the signal to add polynomial features or switch model class.

Test Your Understanding

1. For our 6-sample anchor, verify that the sum of residuals $\sum {\epsilon }_i=0$. Is this always true for OLS? Why?

2. If you doubled all $y$ values (prices in \2k$incrementsinsteadof$$k$),howwould$w_0$and$w_1$ change? Use the OLS formulas to derive this, not trial and error.

3. The SSE dropped from 596600 to 133.33 with OLS weights. Could you find weights that give SSE = 0? If not, why not? If yes, what would that mean about the data?

4. What is the residual for a house with sq_ft = 1250? Is the model over- or under-predicting?

5. A colleague argues that maximizing $R^2$ is the same as minimizing SSE. Are they correct? Under what conditions would these objectives give different results?