Logistic Regression: Math Intuition

You're a loan officer who just learned that linear regression can't produce valid probabilities for classification. The fix is a function called the sigmoid — but the story doesn't end there. Once you replace the linear output with $\sigma (z)$, you need a new loss function (MSE doesn't work for probabilities), a new way to interpret the coefficients (log-odds rather than raw scores), and a gradient that connects all the pieces. Each piece has a formula, and each formula has a concrete number attached to it.

This post traces the full path from raw score to gradient update on six loan applications. Every formula lands on a specific number from this dataset — nothing is symbolic.

What Logistic Regression Does

Logistic regression takes a linear score $z=w_0+w_1\times \text{income}$ and passes it through the sigmoid function $\sigma (z)=1/(1+e^{-z})$ to produce a probability. The coefficients are interpreted in log-odds: a unit change in a feature multiplies the odds of the outcome by $e^w$. The loss function — binary cross-entropy — penalizes wrong confidence, not wrong predictions. And the gradient of this loss with respect to the weights has exactly the same form as linear regression's gradient, but with probabilities in place of raw predictions.

The Plan — Four Steps from Linear Score to Gradient Update

We'll start with a single linear score and convert it to a probability using the sigmoid. We'll reinterpret that probability as log-odds to understand what the coefficients mean. We'll compute the cross-entropy loss to see why MSE is wrong. And finally we'll run one gradient descent step to see how the weights update. Each step is computed on the same six samples.

Anchor dataset: Loan default prediction (6 samples for hand-trace clarity).

import numpy as np

X = np.array([25, 32, 45, 75, 95, 110]).reshape(-1, 1)
y = np.array([1,   1,  1,  0,  0,   0])

# Weights after fitting (used for trace — not starting weights):
# w₀ = 8.12, w₁ = -0.094

Step 1: Linear Score → Sigmoid → Probability

The raw linear score is computed the same way as in linear regression:

$z=w_0+w_1\times \text{income}$

The sigmoid function converts this to a probability:

$\sigma (z)=\frac{1}{1+e^{-z}},\sigma (z)\in (0,1)$

The model predicts $P(y=1|x)=\sigma (z)$, which is the probability of default given income.

Trace for all 6 samples with $w_0=8.12$, $w_1=-0.094$:

At decision threshold 0.5: income = 75 gives $\sigma =0.745$ — predicted as default (wrong). These weights are illustrative; the true MLE solution would correctly separate this dataset.

Green dots are correctly classified; the red dot at income=75k sits above the 0.5 line but is a non-defaulter (y=0). The S-shape ensures all outputs stay within (0, 1).

✓ Step 1 complete. The linear score $z=w_0+w_1\times \text{income}$ ranges from 5.77 (income=25k) to −2.22 (income=110k). After sigmoid, each maps to a valid probability in $(0,1)$. The S-shaped curve guarantees that no input produces an output outside the probability range.

Step 2: Log-Odds (Logit) Interpretation

The ratio of default probability to non-default probability is the odds:

$\text{odds}=\frac{P(y=1)}{P(y=0)}=\frac{\sigma (z)}{1-\sigma (z)}$

Taking the log of odds recovers the linear score exactly:

$\log \left(\frac{P(y=1)}{1-P(y=1)}\right)=z=w_0+w_1\times \text{income}$

The log-odds (logit) is linear in the features. This means logistic regression is a linear model — it draws a straight boundary in feature space — just applied to log-odds rather than probability directly.

Log-odds trace for 3 anchor samples:

Interpreting $w_1=-0.094$: Each additional $1k in income changes the log-odds of default by $-0.094$. In odds terms, it multiplies the odds of default by:

$e^{-0.094}=0.910$

A 9% reduction in the odds of default for each $1k of additional income.

Log-odds is linear in income. The boundary income (z=0) is where the line crosses zero — every sample to the left has positive log-odds (predicted default), every sample to the right has negative log-odds (predicted no default).

✓ Step 2 complete. The log-odds equals the linear score $z$: $\log (p/(1-p))=w_0+w_1\times \text{income}$. Each additional $1kofincomemultipliestheoddsofdefaultby$e^{-0.094} = 0.91$ — a 9% reduction per unit. Logistic regression IS a linear model in log-odds space.

Step 3: Binary Cross-Entropy Loss

Why not MSE? Consider predicting $y=1$ with $\hat{p}=0.999$ (correct, very confident). MSE loss = $(1-0.999{)}^2=0.000001$ — nearly zero. Now predict with $\hat{p}=0.501$ (correct, barely). MSE = $(1-0.501{)}^2=0.249$. The nearly-random prediction is penalized 250,000× more than the confident correct one — backward.

Binary cross-entropy (BCE) penalizes wrong confidence, not wrong predictions:

$L=-\left[y\log (p)+(1-y)\log (1-p)\right]$

Three cases:

• Correct and confident: $y=1$, $p=0.999$ → $L=-\log (0.999)=0.001$ (tiny)
• Correct and unconfident: $y=1$, $p=0.51$ → $L=-\log (0.51)=0.675$ (meaningful signal)
• Wrong and confident: $y=1$, $p=0.001$ → $L=-\log (0.001)=6.908$ (large penalty, $\log$ → $\infty$)

Per-sample loss for the 6-sample anchor:

The sample at income=75 dominates the loss (1.367) because the model is confidently wrong: it predicts $p=0.745$ (likely default) for a non-defaulter. This is exactly where gradient descent will push the decision boundary.

Left panel ($y=1$): loss approaches infinity as $p\to 0$ (confidently wrong). Right panel ($y=0$): loss approaches infinity as $p\to 1$. In both cases, confident correct predictions have loss near zero.

✓ Step 3 complete. Binary cross-entropy penalizes wrong confidence, not wrong predictions. The sample at income=75k dominates the total loss (1.367 of 0.311 average) because the model is confidently wrong — exactly where the gradient needs to push hardest.

Step 4: Gradient Descent Update

The total cost over $n$ samples:

$J(\mathbf{w})=-\frac{1}{n}{\sum }_{i=1}^n\left[y_i\log (p_i)+(1-y_i)\log (1-p_i)\right]$

The gradients work out elegantly — the same form as linear regression but with sigmoid probabilities:

$\frac{\partial J}{\partial w_0}=\frac{1}{n}{\sum }_{i=1}^n(p_i-y_i)$

$\frac{\partial J}{\partial w_1}=\frac{1}{n}{\sum }_{i=1}^n{x}_i(p_i-y_i)$

One gradient step from $w_0=0$, $w_1=0$, $\alpha =0.01$:

With all weights zero: $z=0$ for every sample, so $p_i=\sigma (0)=0.5$.

Prediction errors $p_i-y_i$:

Computing gradients:

$\frac{\partial J}{\partial w_0}=\frac{-0.5-0.5-0.5+0.5+0.5+0.5}{6}=\frac{0}{6}=0.0$

$\frac{\partial J}{\partial w_1}=\frac{25(-0.5)+32(-0.5)+45(-0.5)+75(0.5)+95(0.5)+110(0.5)}{6}$

$=\frac{-12.5-16.0-22.5+37.5+47.5+55.0}{6}=\frac{89.0}{6}=14.83$

Weight updates:

$w_0\leftarrow 0.0-0.01\times 0.0=0.0$

$w_1\leftarrow 0.0-0.01\times 14.83=-0.1483$

The $w_0$ gradient is zero because the dataset is balanced (3 defaulters, 3 non-defaulters) — symmetry cancels. The $w_1$ gradient is positive (14.83) because higher incomes are associated with non-default ($y=0$), so the gradient pushes $w_1$ negative — exactly right. After this one step, the model already knows to decrease $P(\text{default})$ as income increases.

Before (flat line): all probabilities = 0.5, model has no information. After one step: the curve tilts — low income → higher P(default), high income → lower P(default). The direction is correct on step one.

✓ Step 4 complete. From $w_0=0,w_1=0$ with all $p_i=0.5$, one gradient step gives $w_0=0.0,w_1=-0.1483$. The negative $w_1$ already knows that higher income should decrease default probability — correct direction from a single update.

Key Formulas Reference

Related Concepts

The BCE gradient $\partial J/\partial w_j=(1/n)\sum x_i(p_i-y_i)$ has exactly the same structure as the OLS gradient for linear regression — the only difference is $p_i$ (from sigmoid) instead of ${\hat{y}}_i$ (from the linear prediction). This is not a coincidence: both emerge from the chain rule applied through the output activation. Understanding this post requires comfort with the chain rule and what makes a function convex. It unlocks regularization (appending $\frac{\lambda }{2}\Vert w{\Vert }^2$ to $J$ adds $\lambda w_j$ to each gradient), multiclass softmax (the same BCE structure generalizes to $K$ classes), and neural network training (replace sigmoid with other activations, BCE becomes cross-entropy over softmax output).

Honest Limitations

At initialization $w=0$, every sample has $p=0.5$. Gradient descent is guaranteed to find the global minimum because BCE is convex in the weights — the loss surface has no local minima. This property disappears the moment you stack layers: a two-layer network with sigmoid activations is no longer convex. A second limitation: MLE for logistic regression does not have a finite solution when the data is perfectly linearly separable. The weights diverge toward infinity, trying to push all probabilities to exactly 0 or 1. sklearn's default L2 regularization (C=1.0) prevents divergence by adding a penalty, but it changes the optimization from pure MLE to a penalized version — the resulting weights are biased toward zero.

Test Your Understanding

1. The gradient $\partial J/\partial w_0=0$ at initialization because the dataset is balanced. If you added one more defaulter (making 4 defaulters, 3 non-defaulters), what sign would $\partial J/\partial w_0$ have, and what does that mean for $w_0$'s update?

2. At income=75, the loss is 1.367 — the largest single-sample loss. After one gradient step ($w_1=-0.1483$), compute the new $z$ for income=75 and the new $\sigma (z)$. Did the loss for this sample decrease?

3. The gradient of BCE with respect to weights has the form $(1/n)\sum x_i(p_i-y_i)$ — identical in structure to the linear regression gradient. Why does the same form emerge from a completely different loss function?

4. At $w_1=-0.094$ (the approximate fitted weights), the decision boundary (income where $P=0.5$) is at $z=0\Rightarrow w_0+w_{1}x=0\Rightarrow x=-w_0/w_1$. Compute this boundary income value. Does it match your intuition from the data?

5. The log-odds interpretation says each $1k income multiplies the odds of default by $e^{-0.094}=0.91$. If income doubles from 50 to 100, what is the ratio of the odds at 100 to the odds at 50?