Equation of a Line, 3D Plane, and Hyperplane

You're pricing a house — 1,200 square feet, 3 bedrooms. You know bigger houses cost more, and more bedrooms add value. The question is how to combine these two numbers into a single prediction. The answer is a flat surface in math: a line when you use only square footage, a plane when you add bedrooms. Every linear model you'll encounter — logistic regression, the linear layer in a transformer — is doing the same geometric operation in higher-dimensional space. You need to see that shape clearly before you ever train a weight.

This is not a decision tree, which carves the space into rectangular regions. A hyperplane is a single continuous surface — one formula, no if-else branches.

The Equation of a Line (2D)

With one feature — say, square footage — the prediction is a line:

$\hat{y}=w_0+w_{1}x$

$w_0$ is the intercept: the value of $\hat{y}$ when $x=0$. $w_1$ is the slope: how much $\hat{y}$ changes for every one-unit increase in $x$.

For predicting house price from square footage, assume $w_0=50$ and $w_1=0.20$. Every extra square foot adds $0.20k (i.e., $200) to the predicted price.

The two non-zero residuals (at 1100 and 1400 sq ft) tell us the weights aren't quite optimal — but they're close. The goal of training is to find $w_0$ and $w_1$ that minimize the total squared residual.

The slope's sign tells you the direction: $w_1>0$ means larger houses cost more. $w_1<0$ would mean the opposite. $w_1=0$ means the line is horizontal — a feature with no predictive power.

What Changes at 3D: The Equation of a Plane

Add a second feature — number of bedrooms — and the model becomes:

$\hat{y}=w_0+w_1\times \text{sqft}+w_2\times \text{bedrooms}$

With two features, a single prediction now requires values on two axes, and the model surface is a plane floating in 3D. Assume $w_0=30$, $w_1=0.17$, $w_2=15$:

The residuals are larger than the single-feature case — these particular weights ($w_0=30,w_1=0.17,w_2=15$) are illustrative, not optimal. Training will find better values.

Generalizing to p Features: The Hyperplane

With $p$ features the model is:

$\hat{y}=w_0+w_1{x}_1+w_2{x}_2+\cdots +w_p{x}_p$

In compact dot-product form, prepend a 1 to each input vector and absorb the intercept into the weight vector:

$\hat{y}=\mathbf{w}\cdot \mathbf{x}\text{where }\mathbf{w}=[w_0,w_1,\dots ,w_p],\mathbf{x}=[1,x_1,\dots ,x_p]$

A hyperplane in $p+1$ dimensions is still a flat surface — it just can't be visualized beyond 3D. The word "hyper" means dimension, not complexity. The relationship is still linear in the parameters.

The Intercept Trick

Without $w_0$, the hyperplane is forced to pass through the origin. Most real data doesn't pass through the origin — a house with zero square footage doesn't have zero price in the model's internal representation. The standard fix: append a column of ones to the feature matrix.

For the 1-feature anchor, the design matrix $X$ with an intercept column is:

$X=\left[\begin{array}{cc}1& 650\\ 1& 850\\ 1& 1100\\ 1& 1400\\ 1& 1600\\ 1& 1900\end{array}\right],\mathbf{w}=\left[\begin{array}{c}50\\ 0.20\end{array}\right]$

The matrix product $X\mathbf{w}$ gives predictions for all six samples at once:

$X\mathbf{w}=\left[\begin{array}{c}1\times 50+650\times 0.20\\ 1\times 50+850\times 0.20\\ 1\times 50+1100\times 0.20\\ 1\times 50+1400\times 0.20\\ 1\times 50+1600\times 0.20\\ 1\times 50+1900\times 0.20\end{array}\right]=\left[\begin{array}{c}180\\ 220\\ 270\\ 330\\ 370\\ 430\end{array}\right]$

For the 2-feature anchor, $X$ expands to 6×3:

$X=\left[\begin{array}{ccc}1& 650& 2\\ 1& 850& 2\\ 1& 1100& 3\\ 1& 1400& 3\\ 1& 1600& 4\\ 1& 1900& 4\end{array}\right]$

The model $\hat{\mathbf{y}}=X\mathbf{w}$ now holds for all samples simultaneously. This matrix form is how every linear model is implemented at scale — no loops over samples.

Why This Matters for ML

Every linear model is a hyperplane. Logistic regression uses a hyperplane as a decision boundary — points on one side are class 1, the other class 0. SVMs find the hyperplane with maximum margin. The linear layer in a neural network applies this multiplication at each layer. Understanding the geometry now means every subsequent algorithm is just a variation on how the weights $\mathbf{w}$ are found.

The next question is: which $\mathbf{w}$ is best? That requires a loss function.

Geometry Summary

Related Concepts and Honest Limitations

The design matrix $X$ with a leading column of ones is the same representation used to derive the OLS closed-form solution ${\mathbf{w}}^{\ast }=(X^{\top }X{)}^{-1}{X}^{\top }\mathbf{y}$. Understanding why $X^{\top }X$ appears there requires exactly the matrix form developed here. The matrix approach also transfers directly to Ridge regression, where the fix is adding $\lambda I$ to $X^{\top }X$.

A common mistake is assuming more features always make the hyperplane more expressive. The issue isn't expressiveness — it's that when $p$ exceeds $n$, the design matrix is rank-deficient and the weight vector is no longer uniquely determined by the data. The hyperplane exists; there are just infinitely many that fit the training data equally well. This is when regularization stops being optional.

Test Your Understanding

1. With $w_0=50$ and $w_1=0.20$, what is $\hat{y}$ for a house of 1250 sq ft? What is the residual if the true price is $290k?

2. Why does appending a column of ones to $X$ allow the model to learn a non-zero intercept? What would happen geometrically if you left it out and the true intercept was $50k?

3. A colleague proposes fitting two separate lines — one for small houses and one for large houses — instead of a single hyperplane. When would this be better, and what model class formalizes that idea?

4. For the 2-feature case, the coefficient $w_2=15$ means each bedroom adds $15k to price holding sq_ft fixed. How would you confirm this interpretation from the trace table?

5. If you have $p=1000$ features and $n=500$ samples, what does the design matrix $X$ look like, and why does this cause problems for the OLS formula ${\mathbf{w}}^{\ast }=(X^{\top }X{)}^{-1}{X}^{\top }\mathbf{y}$?