Can Linear Regression Solve Classification?

Before learning logistic regression, you should know exactly why linear regression fails at classification. Not "it's not designed for it" — that's circular. The concrete failure modes: predictions that violate probability bounds, a decision boundary that shifts when you add one outlier, and a loss function that cannot discriminate confidence from indecision.

Anchor dataset: Predict loan default (1 = default, 0 = no default) from income.

import numpy as np

# 8 samples: income ($k), y = 1 if default
X = np.array([25, 32, 45, 60, 75, 85, 95, 110]).reshape(-1, 1)
y = np.array([1,   1,  1,  0,  0,  0,  0,   0])
# Pattern: lower income → default

Naive Attempt: Linear Regression on a Binary Target

from sklearn.linear_model import LinearRegression

model = LinearRegression()
model.fit(X, y)
print(f"Intercept: {model.intercept_:.4f}")
print(f"Coef:      {model.coef_[0]:.6f}")

The equation is $\hat{y}=1.6667-0.010417\times \text{income}$. Now compute predictions for each sample:

The model outputs values above 1.0 for four of the eight samples. A probability cannot exceed 1. And for incomes above $160k (extrapolating further), the model would predict a negative probability — equally meaningless.

Red dots are defaulters (y=1), green dots are non-defaulters (y=0). The blue regression line crosses above y=1 for low incomes and would cross below y=0 if we extended to very high incomes. The red dashed lines mark the valid probability range.

The Threshold Hack: Round ŷ to 0 or 1

The obvious fix: apply a threshold. If $\hat{y}>0.5$, predict 1; otherwise predict 0.

Decision boundary: $1.667-0.0104\times x=0.5$ → $x=(1.667-0.5)/0.0104=112.2$

This means the model predicts default for income < $112.2k — which includes every single one of our 8 samples (max income = $110k).

y_pred_thresh = (model.predict(X) > 0.5).astype(int)
print(y_pred_thresh)
# Prediction for all samples:
# Confusion: TP=3 (defaulters called default), FP=5 (non-defaulters called default)
# TN=0, FN=0
accuracy = (y_pred_thresh == y).mean()
print(f"Accuracy: {accuracy:.4f}")
print(f"Baseline (always predict 0): {(y==0).mean():.4f}")

The linear regression classifier achieves 37.5% accuracy — worse than always predicting "no default" (62.5%). The decision boundary landed outside the feature range entirely.

The Outlier Sensitivity Problem

Add one outlier: income = $500k, no default. One wealthy customer should not change how we classify the $25k–$110k range.

X_out = np.vstack([X, [[500]]])
y_out = np.append(y, [0])

model_out = LinearRegression()
model_out.fit(X_out, y_out)
print(f"Original coef: -0.010417")
print(f"New coef:      {model_out.coef_[0]:.6f}")

# New boundary: 1/(new_slope) scale calculation
new_boundary = (model_out.intercept_ - 0.5) / (-model_out.coef_[0])
print(f"New decision boundary: ${new_boundary:.1f}k")

The boundary shifted from $112k to $51k. Samples at $60k, $75k, $85k, $95k, $110k (all non-defaulters) are now predicted as defaulters. Adding one legitimate outlier corrupted the predictions for five correctly-classified samples.

Left panel: correct classification at $112k boundary. Right panel: outlier pulls the regression line down, new boundary at $51k misclassifies five non-defaulters (the green dots now fall in the red zone).

What We Actually Need

The sigmoid function maps any real-valued score to a valid probability:

$\sigma (z)=\frac{1}{1+e^{-z}}$

For any $z\in (-\infty ,+\infty )$, $\sigma (z)\in (0,1)$:

import numpy as np

for z in [-5, -2, 0, 2, 5]:
    s = 1 / (1 + np.exp(-z))
    print(f"σ({z:+d}) = {s:.4f}")

The model becomes $P(y=1|x)=\sigma (w_0+w_1\times \text{income})$. The decision boundary is where $\sigma (z)=0.5$, which means $z=0$, which means $w_0+w_1\times \text{income}=0$ — a well-defined linear equation regardless of the data range.

The outlier sensitivity is fixed because sigmoid squashes extreme values: an income of $500k produces a very large negative $z$, and $\sigma (\text{very negative})\approx 0$ regardless of exactly how negative. Adding one extreme outlier slightly adjusts the weights but doesn't destroy the boundary.

The Three Fundamental Problems

1. Range violation: Linear regression outputs can exceed $[0,1]$ — not interpretable as probabilities. Sigmoid fixes this by construction.

2. Outlier sensitivity: One extreme sample shifts the regression line and displaces the decision boundary, misclassifying an arbitrary number of correctly-handled samples. The sigmoid's saturation at extreme values absorbs outliers gracefully.

3. Wrong loss function: MSE treats the problem as predicting a continuous value. A prediction of 0.999 (correct, confident) and 0.5 (correct, no confidence) have MSE losses of 0.000001 and 0.25 relative to $y=1$. Binary cross-entropy properly assigns a large loss to confident wrong predictions and grows unboundedly — the gradient signal is strong where the model most needs to correct.

Linear vs Logistic Regression for Classification

Test Your Understanding

1. The threshold at 0.5 moved the decision boundary to $112k, which misclassified the entire dataset. If you lowered the threshold to 0.3, what income would become the new boundary? Would accuracy improve?

2. Adding one outlier shifted the boundary from $112k to $51k, misclassifying 5 samples. How would the boundary shift if the outlier had income = $5000k instead of $500k?

3. $\sigma (0)=0.5$ always, regardless of the weights. What does this mean about the decision boundary (the income where P(default) = 0.5) in logistic regression, and how does it differ from the linear regression boundary?

4. The three problems are: range violation, outlier sensitivity, and wrong loss. If you replaced MSE with mean absolute error (MAE) for linear regression classification, which problems would remain?

5. If the data were perfectly linearly separable (a clear income gap between all defaulters and non-defaulters), would linear regression with threshold 0.5 give correct predictions? What breaks the approach even in this ideal case?