Can Linear Regression Solve Classification?

You're a loan officer at a community bank. You have records from eight recent applicants — some repaid their loans, some defaulted. Your manager wants a model: given an applicant's annual income, predict whether they will default. The most natural tool is linear regression — you plug in income, you get a number, and you have a prediction. It works for house prices and sales forecasts, so it should work for this too.

It doesn't. Not in a subtle way — in ways that break the entire exercise. The numbers come back as 1.4, 1.3, and even negative values — impossible as probabilities. The decision boundary lurches when you add one unusual applicant. And the loss function treats a confident correct guess and a barely-better-than-random guess as equally good, as long as both land on the right side of a cutoff.

Each of these failures points at a specific design choice a proper classification model must fix. You'll walk through all three on a concrete set of loan applications — the need for the sigmoid function and binary cross-entropy will feel inevitable rather than invented.

What Linear Regression Does (and Why It's Wrong for Classification)

Linear regression for classification means taking a binary outcome — default = 1, no default = 0 — and fitting the usual straight line through the data points. The prediction equation is $\hat{y}=w_0+w_1\times \text{income}$, just like always. The problem is that $\hat{y}$ is supposed to represent a probability, but nothing in the linear regression setup keeps it inside $[0,1]$. The obvious fix — round to 0 or 1 at some threshold — makes things worse, and the deeper issue is that the loss function (MSE) rewards the wrong behavior for binary targets.

The Plan — Four Steps to Understand Why Linear Regression Fails

We'll trace four failures, each building on the last. First we fit linear regression directly and see predictions that violate probability bounds. We try a threshold hack and watch accuracy drop below a naive baseline. We add one outlier and see the decision boundary shift across the entire dataset. Finally we see the one change that fixes all three problems at once.

Anchor dataset: Predict loan default (1 = default, 0 = no default) from income.

import numpy as np

# 8 samples: income ($k), y = 1 if default
X = np.array([25, 32, 45, 60, 75, 85, 95, 110]).reshape(-1, 1)
y = np.array([1,   1,  1,  0,  0,  0,  0,   0])
# Pattern: lower income → default

Step 1: Naive Attempt — Fit Linear Regression on a Binary Target

from sklearn.linear_model import LinearRegression

model = LinearRegression()
model.fit(X, y)
print(f"Intercept: {model.intercept_:.4f}")
print(f"Coef:      {model.coef_[0]:.6f}")

Intercept: 1.6667
Coef:      -0.010417

The equation is $\hat{y}=1.6667-0.010417\times \text{income}$. Now compute predictions for each sample:

The model outputs values above 1.0 for four of the eight samples. A probability cannot exceed 1. And for incomes above $160k (extrapolating further), the model would predict a negative probability — equally meaningless.

Red dots are defaulters (y=1), green dots are non-defaulters (y=0). The blue regression line crosses above y=1 for low incomes and would cross below y=0 if we extended to very high incomes. The red dashed lines mark the valid probability range.

✓ Step 1 complete. The linear regression line predicts probabilities above 1.0 for four of eight samples and would predict below 0.0 if extrapolated. The output is not interpretable as a probability — the first fundamental failure.

Step 2: The Threshold Hack — Accuracy Drops Below a Naive Baseline

The obvious fix: apply a threshold. If $\hat{y}>0.5$, predict 1; otherwise predict 0.

Decision boundary: $1.667-0.0104\times x=0.5$ → $x=(1.667-0.5)/0.0104=112.2$

This means the model predicts default for income < $112.2k — which includes every single one of our 8 samples (max income = $110k).

y_pred_thresh = (model.predict(X) > 0.5).astype(int)
print(y_pred_thresh)
# Prediction for all samples:
# Confusion: TP=3 (defaulters called default), FP=5 (non-defaulters called default)
# TN=0, FN=0
accuracy = (y_pred_thresh == y).mean()
print(f"Accuracy: {accuracy:.4f}")
print(f"Baseline (always predict 0): {(y==0).mean():.4f}")

[1 1 1 1 1 1 1 1]
Accuracy: 0.3750
Baseline (always predict 0): 0.6250

The linear regression classifier achieves 37.5% accuracy — worse than always predicting "no default" (62.5%). The decision boundary landed outside the feature range entirely.

✓ Step 2 complete. A threshold at 0.5 gives a boundary at $112.2k — classifying every single sample as default. Accuracy = 37.5% is worse than the naive baseline of always predicting non-default (62.5%).

Step 3: The Outlier Sensitivity Problem — One Point Moves the Entire Boundary

Add one outlier: income = $500k, no default. One wealthy customer should not change how we classify the $25k–$110k range.

X_out = np.vstack([X, [[500]]])
y_out = np.append(y, [0])

model_out = LinearRegression()
model_out.fit(X_out, y_out)
print(f"Original coef: -0.010417")
print(f"New coef:      {model_out.coef_[0]:.6f}")

# New boundary: 1/(new_slope) scale calculation
new_boundary = (model_out.intercept_ - 0.5) / (-model_out.coef_[0])
print(f"New decision boundary: ${new_boundary:.1f}k")

Original coef: -0.010417
New coef:      -0.001804
New decision boundary: $51.4k

The boundary shifted from $112k to $51k. Samples at $60k, $75k, $85k, $95k, $110k (all non-defaulters) are now predicted as defaulters. Adding one legitimate outlier corrupted the predictions for five correctly-classified samples.

Left panel: correct classification at $112k boundary. Right panel: outlier pulls the regression line down, new boundary at $51k misclassifies five non-defaulters (the green dots now fall in the red zone).

✓ Step 3 complete. Adding one outlier at $500kshiftsthedecisionboundaryfrom$112k to $51k, misclassifying five previously-correct non-defaulters. One point corrupts the model for the entire low-to-mid income range.

Step 4: What We Actually Need — The Sigmoid Fix

The sigmoid function maps any real-valued score to a valid probability:

$\sigma (z)=\frac{1}{1+e^{-z}}$

For any $z\in (-\infty ,+\infty )$, $\sigma (z)\in (0,1)$:

import numpy as np

for z in [-5, -2, 0, 2, 5]:
    s = 1 / (1 + np.exp(-z))
    print(f"σ({z:+d}) = {s:.4f}")

σ(-5) = 0.0067
σ(-2) = 0.1192
σ( 0) = 0.5000
σ(+2) = 0.8808
σ(+5) = 0.9933

The model becomes $P(y=1|x)=\sigma (w_0+w_1\times \text{income})$. The decision boundary is where $\sigma (z)=0.5$, which means $z=0$, which means $w_0+w_1\times \text{income}=0$ — a well-defined linear equation regardless of the data range.

The outlier sensitivity is fixed because sigmoid squashes extreme values: an income of $500k produces a very large negative $z$, and $\sigma (\text{very negative})\approx 0$ regardless of exactly how negative. Adding one extreme outlier slightly adjusts the weights but doesn't destroy the boundary.

✓ Step 4 complete. The sigmoid function maps any real number to $(0,1)$, fixing the range violation. The decision boundary is always well-defined at $\sigma (z)=0.5\to z=0$. Extreme values are squashed, making the fit robust to outliers.

The Three Fundamental Problems

1. Range violation: Linear regression outputs can exceed $[0,1]$ — not interpretable as probabilities. Sigmoid fixes this by construction.

2. Outlier sensitivity: One extreme sample shifts the regression line and displaces the decision boundary, misclassifying an arbitrary number of correctly-handled samples. The sigmoid's saturation at extreme values absorbs outliers gracefully.

3. Wrong loss function: MSE treats the problem as predicting a continuous value. A prediction of 0.999 (correct, confident) and 0.5 (correct, no confidence) have MSE losses of 0.000001 and 0.25 relative to $y=1$. Binary cross-entropy properly assigns a large loss to confident wrong predictions and grows unboundedly — the gradient signal is strong where the model most needs to correct.

Linear vs Logistic Regression for Classification

Related Concepts

Linear regression works here by minimizing squared error, which assumes residuals are roughly continuous and symmetric — a binary 0/1 target violates both assumptions. Logistic regression is the direct fix: replace the output with a sigmoid, replace MSE with binary cross-entropy, and the same gradient descent machinery applies. From here, the natural extensions are L1/L2 regularization (add a penalty term to the BCE loss and update the gradient) and multiclass problems (OvR or softmax).

Honest Limitations

The outlier fragility is real but specific: linear regression's decision boundary shifts because the OLS coefficient is a ratio of covariances, which is sensitive to leverage points. If every feature value were bounded within a known range and the data were perfectly linearly separable with no overlap, a threshold applied to a linear regression model would give correct classifications — but the outputs would still not be interpretable as probabilities. In practice, even mild class overlap breaks this edge case. The probability bound violation (ŷ > 1 or ŷ < 0) is not just aesthetic — downstream systems that expect calibrated probabilities (e.g., risk scoring, Platt scaling) will fail on unbound outputs regardless of classification accuracy.

Test Your Understanding

1. The threshold at 0.5 moved the decision boundary to $112k, which misclassified the entire dataset. If you lowered the threshold to 0.3, what income would become the new boundary? Would accuracy improve?

2. Adding one outlier shifted the boundary from $112k to $51k, misclassifying 5 samples. How would the boundary shift if the outlier had income = $5000k instead of $500k?

3. $\sigma (0)=0.5$ always, regardless of the weights. What does this mean about the decision boundary (the income where P(default) = 0.5) in logistic regression, and how does it differ from the linear regression boundary?

4. The three problems are: range violation, outlier sensitivity, and wrong loss. If you replaced MSE with mean absolute error (MAE) for linear regression classification, which problems would remain?

5. If the data were perfectly linearly separable (a clear income gap between all defaulters and non-defaulters), would linear regression with threshold 0.5 give correct predictions? What breaks the approach even in this ideal case?