SVM: Hard Margin and Soft Margin

Logistic regression stops when the loss is low. Given a linearly separable dataset, infinitely many hyperplanes achieve zero training loss — logistic regression picks whichever one gradient descent reaches first. SVM asks a harder question: among all separating hyperplanes, which one is the most confident? The answer is the hyperplane that maximizes the gap between the two classes — the maximum-margin classifier.

Anchor dataset: Binary loan default prediction from income and credit score.

import numpy as np
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler

# 8 samples: [income ($k), credit_score_normalized]
X = np.array([
    [25, 0.3],  [30, 0.4],  [40, 0.5],  [55, 0.6],   # default (y=-1)
    [70, 0.7],  [80, 0.8],  [90, 0.8],  [100, 0.9],  # no default (y=+1)
])
y = np.array([-1, -1, -1, -1, 1, 1, 1, 1])  # SVM uses +1/-1 labels

The Motivation: Why Maximize the Margin?

Any separating hyperplane correctly classifies the training data, but some are far more fragile than others. A hyperplane that passes close to the nearest points gives almost no room for new data that differs slightly from training — a 5% deviation in income or credit score would flip the prediction.

The maximum-margin hyperplane maximizes the distance to the nearest training points from each class. A wider margin means more tolerance for perturbation in new data, which is why SVM tends to generalize better than logistic regression on small datasets.

The green solid line is the SVM solution — equidistant from both classes. The two green dashed lines are the margin boundaries ($w\cdot x+b=\pm 1$). The gap labeled $2/\Vert w\Vert$ is the margin width.

Hard Margin SVM — Math Setup

The separating hyperplane is $w\cdot x+b=0$. Two parallel planes define the margin:

$w\cdot x+b=+1\text{(positive margin boundary)}$ $w\cdot x+b=-1\text{(negative margin boundary)}$

From post 03 (distance of a point from a plane): the distance from the origin plane to the $+1$ plane is $|+1-0|/\Vert w\Vert =1/\Vert w\Vert$. By symmetry, the distance to the $-1$ plane is also $1/\Vert w\Vert$. The total margin is:

$\text{margin}=\frac{2}{\Vert w\Vert }$

Maximizing $2/\Vert w\Vert$ is equivalent to minimizing $\Vert w{\Vert }^2/2$ (a differentiable, convex objective).

Hard margin primal problem:

$\text{Minimize }\frac{1}{2}\Vert w{\Vert }^2\text{subject to }{y}_i(w\cdot x_i+b)\ge 1\text{ for all }i$

The combined constraint $y_i(w\cdot x_i+b)\ge 1$ encodes both classes simultaneously: for $y_i=+1$, it requires $w\cdot x_i+b\ge 1$; for $y_i=-1$, it requires $w\cdot x_i+b\le -1$.

Identifying Support Vectors

Support vectors are the training points that lie exactly on the margin boundaries — the closest points to the hyperplane. They "support" the margin: if you removed any other point, the margin wouldn't change.

Using approximate weights $w=[0.08,2.5]$, $b=-6.5$ for the anchor:

Samples at income=55 and income=70 have $y_i{f}_i<1$ — they violate the hard margin constraint. This shows that these approximate weights aren't the true optimum; the exact QP solution adjusts weights until the nearest samples from each class sit exactly at $y_i{f}_i=1$.

Soft Margin SVM — Allowing Violations

Hard margin fails when:

1. The data is not linearly separable (classes overlap)
2. A single outlier prevents any hard-margin solution

Soft margin introduces slack variables ${\xi }_i\ge 0$ — one per sample — measuring how much the margin constraint is violated:

• ${\xi }_i=0$: point correctly classified and outside the margin ✓
• $0<{\xi }_i<1$: point inside the margin but on the correct side
• ${\xi }_i\ge 1$: point on the wrong side of the hyperplane (misclassified)

Soft margin primal problem:

$\text{Minimize}\frac{1}{2}\Vert w{\Vert }^2+C{\sum }_i{\xi }_i$

$\text{subject to}{y}_i(w\cdot x_i+b)\ge 1-{\xi }_i,{\xi }_i\ge 0\text{for all }i$

The hyperparameter $C$ controls the tradeoff between margin width and constraint violations.

The C Hyperparameter

scaler = StandardScaler()
X_sc = scaler.fit_transform(X)

C_values = [0.01, 0.1, 1, 10, 100]
for C in C_values:
    svm = SVC(kernel='linear', C=C)
    svm.fit(X_sc, y)
    n_sv = svm.support_vectors_.shape[0]
    margin = 2 / np.linalg.norm(svm.coef_)
    print(f"C={C:>6}: n_support_vectors={n_sv}, margin=2/‖w‖={margin:.4f}")

As $C$ increases: fewer support vectors (the boundary hugs the training data more tightly), narrower margin. As $C$ decreases: more support vectors, wider margin.

Left: wide margin at $C=0.01$ — points inside the margin are allowed. Center: balanced margin at $C=1$ with 3 support vectors (orange circles). Right: narrow margin at $C=100$ with only 2 support vectors — the boundary is tight.

Support Vectors Are All That Matter

After training, prediction for a new point $x_{\text{new}}$ depends only on the support vectors:

$f(x_{\text{new}})={\sum }_{i\in \text{SV}}{\alpha }_i{y}_i(x_i\cdot x_{\text{new}})+b$

This is the dual form of SVM. The ${\alpha }_i$ are Lagrange multipliers — non-zero only for support vectors (all other ${\alpha }_i=0$). This sparsity is what makes SVM efficient at prediction time and enables the kernel trick (post 03).

Hard Margin vs Soft Margin

Test Your Understanding

1. The margin is $2/\Vert w\Vert$. If you scale all features by a constant factor $k$ (i.e., multiply $X$ by $k$), what happens to $\Vert w\Vert$ and the margin? Does the decision boundary change?

2. The hard margin constraint is $y_i(w\cdot x_i+b)\ge 1$. If you rescale the constraint to $y_i(w\cdot x_i+b)\ge 2$, does this change the problem? Is the resulting hyperplane different?

3. At $C=0.01$, there are 7 support vectors (out of 8 training samples). At $C=100$, there are only 2. Qualitatively, what happens to the model's bias and variance as $C$ decreases from 100 to 0.01?

4. The slack variable ${\xi }_i$ for income=55 is nonzero (the sample is inside the margin). If you removed that sample from the training set, would the optimal hyperplane change? How do you know?

5. In the dual form $f(x)={\sum }_{i\in \text{SV}}{\alpha }_i{y}_i(x_i\cdot x)+b$, if two support vectors from the positive class ($y=+1$) are at $x_1=[55,0.6]$ and $x_2=[70,0.7]$, compute the contribution of each to $f([60,0.65])$ (ignoring ${\alpha }_i$ and $b$).