Logistic Regression: Math Intuition

Logistic regression produces probabilities, not class labels. Understanding where those probabilities come from — and why binary cross-entropy is the right loss — requires tracing the full path from raw linear score to gradient update. Every formula here is computed on concrete numbers.

Anchor dataset: Loan default prediction (6 samples for hand-trace clarity).

import numpy as np

X = np.array([25, 32, 45, 75, 95, 110]).reshape(-1, 1)
y = np.array([1,   1,  1,  0,  0,   0])

# Weights after fitting (used for trace — not starting weights):
# w₀ = 8.12, w₁ = -0.094

Step 1: Linear Score → Sigmoid → Probability

The raw linear score is computed the same way as in linear regression:

$z=w_0+w_1\times \text{income}$

The sigmoid function converts this to a probability:

$\sigma (z)=\frac{1}{1+e^{-z}},\sigma (z)\in (0,1)$

The model predicts $P(y=1|x)=\sigma (z)$, which is the probability of default given income.

Trace for all 6 samples with $w_0=8.12$, $w_1=-0.094$:

At decision threshold 0.5: income = 75 gives $\sigma =0.745$ — predicted as default (wrong). These weights are illustrative; the true MLE solution would correctly separate this dataset.

Green dots are correctly classified; the red dot at income=75k sits above the 0.5 line but is a non-defaulter (y=0). The S-shape ensures all outputs stay within (0, 1).

Step 2: Log-Odds (Logit) Interpretation

The ratio of default probability to non-default probability is the odds:

$\text{odds}=\frac{P(y=1)}{P(y=0)}=\frac{\sigma (z)}{1-\sigma (z)}$

Taking the log of odds recovers the linear score exactly:

$\log \left(\frac{P(y=1)}{1-P(y=1)}\right)=z=w_0+w_1\times \text{income}$

The log-odds (logit) is linear in the features. This means logistic regression is a linear model — it draws a straight boundary in feature space — just applied to log-odds rather than probability directly.

Log-odds trace for 3 anchor samples:

Interpreting $w_1=-0.094$: Each additional $1k in income changes the log-odds of default by $-0.094$. In odds terms, it multiplies the odds of default by:

$e^{-0.094}=0.910$

A 9% reduction in the odds of default for each $1k of additional income.

Step 3: Binary Cross-Entropy Loss

Why not MSE? Consider predicting $y=1$ with $\hat{p}=0.999$ (correct, very confident). MSE loss = $(1-0.999{)}^2=0.000001$ — nearly zero. Now predict with $\hat{p}=0.501$ (correct, barely). MSE = $(1-0.501{)}^2=0.249$. The nearly-random prediction is penalized 250,000× more than the confident correct one — backward.

Binary cross-entropy (BCE) penalizes wrong confidence, not wrong predictions:

$L=-\left[y\log (p)+(1-y)\log (1-p)\right]$

Three cases:

• Correct and confident: $y=1$, $p=0.999$ → $L=-\log (0.999)=0.001$ (tiny)
• Correct and unconfident: $y=1$, $p=0.51$ → $L=-\log (0.51)=0.675$ (meaningful signal)
• Wrong and confident: $y=1$, $p=0.001$ → $L=-\log (0.001)=6.908$ (large penalty, $\log$ → $\infty$)

Per-sample loss for the 6-sample anchor:

The sample at income=75 dominates the loss (1.367) because the model is confidently wrong: it predicts $p=0.745$ (likely default) for a non-defaulter. This is exactly where gradient descent will push the decision boundary.

Left panel ($y=1$): loss approaches infinity as $p\to 0$ (confidently wrong). Right panel ($y=0$): loss approaches infinity as $p\to 1$. In both cases, confident correct predictions have loss near zero.

Step 4: Gradient Descent Update

The total cost over $n$ samples:

$J(\mathbf{w})=-\frac{1}{n}{\sum }_{i=1}^n\left[y_i\log (p_i)+(1-y_i)\log (1-p_i)\right]$

The gradients work out elegantly — the same form as linear regression but with sigmoid probabilities:

$\frac{\partial J}{\partial w_0}=\frac{1}{n}{\sum }_{i=1}^n(p_i-y_i)$

$\frac{\partial J}{\partial w_1}=\frac{1}{n}{\sum }_{i=1}^n{x}_i(p_i-y_i)$

One gradient step from $w_0=0$, $w_1=0$, $\alpha =0.01$:

With all weights zero: $z=0$ for every sample, so $p_i=\sigma (0)=0.5$.

Prediction errors $p_i-y_i$:

Computing gradients:

$\frac{\partial J}{\partial w_0}=\frac{-0.5-0.5-0.5+0.5+0.5+0.5}{6}=\frac{0}{6}=0.0$

$\frac{\partial J}{\partial w_1}=\frac{25(-0.5)+32(-0.5)+45(-0.5)+75(0.5)+95(0.5)+110(0.5)}{6}$

$=\frac{-12.5-16.0-22.5+37.5+47.5+55.0}{6}=\frac{89.0}{6}=14.83$

Weight updates:

$w_0\leftarrow 0.0-0.01\times 0.0=0.0$

$w_1\leftarrow 0.0-0.01\times 14.83=-0.1483$

The $w_0$ gradient is zero because the dataset is balanced (3 defaulters, 3 non-defaulters) — symmetry cancels. The $w_1$ gradient is positive (14.83) because higher incomes are associated with non-default ($y=0$), so the gradient pushes $w_1$ negative — exactly right. After this one step, the model already knows to decrease $P(\text{default})$ as income increases.

Key Formulas Reference

Test Your Understanding

1. The gradient $\partial J/\partial w_0=0$ at initialization because the dataset is balanced. If you added one more defaulter (making 4 defaulters, 3 non-defaulters), what sign would $\partial J/\partial w_0$ have, and what does that mean for $w_0$'s update?

2. At income=75, the loss is 1.367 — the largest single-sample loss. After one gradient step ($w_1=-0.1483$), compute the new $z$ for income=75 and the new $\sigma (z)$. Did the loss for this sample decrease?

3. The gradient of BCE with respect to weights has the form $(1/n)\sum x_i(p_i-y_i)$ — identical in structure to the linear regression gradient. Why does the same form emerge from a completely different loss function?

4. At $w_1=-0.094$ (the approximate fitted weights), the decision boundary (income where $P=0.5$) is at $z=0\Rightarrow w_0+w_{1}x=0\Rightarrow x=-w_0/w_1$. Compute this boundary income value. Does it match your intuition from the data?

5. The log-odds interpretation says each $1k income multiplies the odds of default by $e^{-0.094}=0.91$. If income doubles from 50 to 100, what is the ratio of the odds at 100 to the odds at 50?