Multiclass Logistic Regression: OvR (One vs Rest)

Your logistic regression model works well for two classes — default vs no default, malignant vs benign. But your new dataset has three iris species to classify: Setosa, Versicolor, and Virginica. You can't ask "probability of class 1 vs class 0" when there are three possibilities. You need a strategy that turns one binary classifier into many.

One-vs-Rest (OvR) is the simplest: train one binary classifier per class, each asking "Is this sample in my class, or not?" At prediction time, run all classifiers and pick the one with the highest confidence score. The catch is that three independent classifiers produce three independent scores — and those scores don't sum to 1 like probabilities should. Understanding this weakness is the key to knowing when OvR works and when to reach for softmax instead.

What OvR Does

OvR decomposes a $K$-class problem into $K$ binary problems. For class $k$, relabel all samples: $y=1$ if the sample belongs to class $k$, $y=0$ otherwise. Train a standard binary logistic regression on this relabeled dataset. Repeat for $k=1\dots K$. At prediction time, compute $\sigma (w_k\cdot x)$ for each classifier, normalize the $K$ scores by their sum, and predict $\arg \max$. The key insight: each classifier sees the full dataset (5/6 of it as negative), so the class imbalance per classifier is severe — one classifier might have only 2 positives and 4 negatives, while another has 1 positive and 5 negatives.

The Plan — Five Steps from Binary to Multiclass via OvR

We'll start by understanding why binary logistic regression can't handle 3 classes directly. Then we'll train 3 binary classifiers on a 6-sample Iris subset and trace their per-class probabilities. We'll see the key weakness — probabilities that don't sum to 1 — and fix it with normalization. Finally we'll implement the full OvR pipeline in sklearn.

Anchor dataset: Iris flowers — petal length and petal width classify 3 species.

from sklearn.datasets import load_iris
import numpy as np

iris = load_iris()
# Classes: 0=Setosa, 1=Versicolor, 2=Virginica
# Features used: petal_length (col 2), petal_width (col 3)

# 6-sample hand-trace subset (2 per class)
X_trace = np.array([
    [1.4, 0.2],  # Setosa
    [1.5, 0.4],  # Setosa
    [4.7, 1.4],  # Versicolor
    [4.5, 1.5],  # Versicolor
    [6.1, 2.3],  # Virginica
    [5.8, 1.8],  # Virginica
])
y_trace = np.array([0, 0, 1, 1, 2, 2])

Step 1: Why Binary Logistic Regression Can't Handle 3 Classes

Logistic regression is binary by design: it models $P(y=1|x)$ versus $P(y=0|x)$. Two extension strategies:

• One-vs-Rest (OvR): train $K$ binary classifiers (one per class). Each is fit independently. At prediction time, run all $K$ and pick the class with highest confidence.
• Softmax (Multinomial): train one joint classifier that directly outputs $K$ probabilities summing to 1.

OvR is simpler, works with any binary classifier, and is sklearn's default for logistic regression.

✓ Step 1 complete. Binary logistic regression draws one decision boundary. With $K=3$ classes, one boundary is insufficient — the natural extension is to train $K$ binary classifiers, each separating one class from the rest.

Step 2: Train 3 Binary Classifiers — Relabeling the Data

For each class, relabel the 6-sample anchor: the class of interest becomes 1, all others become 0.

Classifier 1 — Setosa vs {Versicolor, Virginica}:

Classifiers 2 and 3 use the same table with the y column relabeled: Versicolor=1 for Classifier 2, Virginica=1 for Classifier 3.

Approximate weights learned by sklearn (stated, not hand-derived):

• Classifier 1 (Setosa): $\mathbf{w}=[w_0=-6.2,w_1=-2.8,w_2=-4.1]$
• Classifier 2 (Versicolor): $\mathbf{w}=[w_0=0.4,w_1=0.9,w_2=-1.2]$
• Classifier 3 (Virginica): $\mathbf{w}=[w_0=-5.1,w_1=1.9,w_2=3.3]$

✓ Step 2 complete. Three binary classifiers trained, each with its own set of weights. The Versicolor classifier gives its class positive weight ($w_1=0.9$) and the Virginica feature a negative weight ($w_2=-1.2$), reflecting the overlap in petal dimensions.

Step 3: Per-Class Probability Trace — Versicolor-1

Compute $z=w_0+w_1\times \text{petal\_l}+w_2\times \text{petal\_w}$ and $\sigma (z)$ for each classifier on sample Versicolor-1 (petal_l=4.7, petal_w=1.4):

Decision: argmax of [0.000, 0.950, 1.000] → Virginica (wrong — true class is Versicolor).

✓ Step 3 complete. Sample Versicolor-1 (petal_l=4.7, petal_w=1.4) gets σ(Versicolor)=0.950 but σ(Virginica)≈1.000. The Virginica classifier overpowers the correct one because its boundary was trained only to distinguish "large petal vs not" — it can't distinguish Versicolor from Virginica specifically.

Step 4: OvR's Key Weakness — Probabilities Don't Sum to 1

The three sigmoid values sum to $0.000+0.950+1.000=1.950$, not 1. Each classifier is trained independently without knowledge of the others — there's no constraint enforcing that the probabilities are collectively coherent.

Sklearn normalizes by dividing each by the sum:

$\hat{p}(\text{Setosa})=0/1.95=0.000$ $\hat{p}(\text{Versicolor})=0.95/1.95=0.487$ $\hat{p}(\text{Virginica})=1.00/1.95=0.513$

Final prediction: Virginica (0.513 > 0.487). Even after normalization, Versicolor-1 is still misclassified — Classifier 3 (Virginica) assigns a score of 1.000 because the Virginica vs {Setosa, Versicolor} boundary places many Versicolor samples on the Virginica side.

Full 6-sample prediction table:

Versicolor is the hard class — its petal dimensions overlap with Virginica. The 2D feature space (petal_l vs petal_w) doesn't fully separate these two species, and the Virginica classifier (trained to detect anything that's not Setosa or Versicolor) picks up large-petal Versicolor samples.

The two Versicolor samples (circled in red) sit in the Virginica decision region because Classifier 3 draws a boundary that encloses large-petal samples regardless of species.

✓ Step 4 complete. Three raw sigmoid scores sum to 1.950 — not 1. Normalization fixes this arithmetically but doesn't fix the root cause: independent classifiers have no shared loss to enforce coherent probability estimates.

Step 5: sklearn OvR Implementation

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris

iris = load_iris()
X, y = iris.data[:, 2:4], iris.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc  = scaler.transform(X_test)

model = LogisticRegression(multi_class='ovr', solver='lbfgs', max_iter=1000)
model.fit(X_train_sc, y_train)

print("Coefficients per class:")
for i, cls in enumerate(iris.target_names):
    print(f"  {cls:12s}: {model.coef_[i].round(3)}")

print(f"\nTest accuracy: {model.score(X_test_sc, y_test):.4f}")

Coefficients per class:
  setosa      : [-1.432 -1.095]
  versicolor  : [ 0.582 -0.421]
  virginica   : [ 0.850  1.516]

Test accuracy: 0.9667

model.coef_ has shape (K, p) — one row per class. The Virginica classifier has a large positive coefficient for petal_width (1.516) because wider petals strongly predict Virginica.

✓ Step 5 complete. sklearn's LogisticRegression(multi_class='ovr') trains $K=3$ binary classifiers and normalizes scores. Test accuracy = 96.67% on the Iris dataset using only petal features.

OvR vs Softmax (Multinomial)

The key architectural difference: OvR classifiers share no information during training. Classifier 1 doesn't know that Classifier 3 will claim the same region. Softmax solves a joint optimization where the sum constraint is enforced during training — better calibrated probabilities but requires that your model can output probabilities (logistic regression can; SVMs cannot without calibration).

OvR Prediction Rule

1. Train $K$ binary classifiers (one per class)
2. For new sample $x$: compute $\sigma (w_k\cdot x)$ for $k=1\dots K$
3. Normalize: ${\hat{p}}_k={\sigma }_k/{\sum }_i{\sigma }_i$
4. Predict: $\hat{y}=\arg {\max }_k{\hat{p}}_k$

Related Concepts

OvR extends binary logistic regression (Post 02) to $K$ classes by decomposing the problem into $K$ independent binary decisions — each classifier uses the same sigmoid, BCE loss, and gradient descent. The direct alternative is softmax (multinomial) logistic regression, which optimizes a joint objective over all classes; softmax reappears as the standard output activation in neural network classifiers. The "train K classifiers and pick the max" pattern also shows up in multiclass SVMs (also OvR by default) and one-vs-one schemes (K(K−1)/2 classifiers, majority vote).

Honest Limitations

The core OvR limitation is that classifiers trained independently have no knowledge of each other's boundaries. When two classes are close in feature space — as Versicolor and Virginica are in petal dimensions — neither classifier has a strong discriminative signal between those two specifically: Classifier 2 (Versicolor vs others) and Classifier 3 (Virginica vs others) both learn primarily from Setosa samples, which are far from the Versicolor/Virginica boundary. Softmax handles ambiguous pairs better because it optimizes a joint loss that sees all classes simultaneously. A second limitation: at inference time, OvR requires $K$ separate forward passes. For K=1000 classes (ImageNet-scale) this is 1000 sigmoid evaluations; softmax is a single matrix multiply — one pass regardless of K.

Test Your Understanding

1. The sum of raw OvR probabilities for Versicolor-1 is 1.950. If you added a fourth class (Iris setosa hybrid) with a classifier outputting $\sigma =0.3$ for this sample, would the final prediction (after normalization) still be Virginica?

2. OvR trains K=3 binary classifiers on a dataset with n=150 samples. Each classifier trains on all 150 samples (just with relabeled y). How does the class imbalance differ across the 3 binary problems? Which classifier sees the most severe imbalance?

3. The OvR test accuracy is 96.67%. The two Versicolor samples were misclassified in our 6-sample trace. Are these same samples likely misclassified on the full 150-sample model? Why or why not?

4. You train OvR on a 10-class problem with 5,000 samples. How many total binary classifiers are trained, and what is the size of the training set (with labels) for each?

5. Softmax guarantees probabilities sum to 1 by construction. If OvR's normalized probabilities are ≈[0.0, 0.487, 0.513] for a sample, what additional information would Softmax's training use that OvR ignores?