Support Vector Machines

Imagine you're building a loan approval system. You have historical data on applicants — income and credit score — labeled as default or no default. You train a logistic regression classifier, and it finds a separating line between the two classes. But here's the uncomfortable truth: that line isn't unique. Given a linearly separable dataset, infinitely many lines achieve zero training loss. Logistic regression just picks the first one gradient descent lands on, with no guarantee it's the one that will generalize best to tomorrow's applicants.

Now someone new applies with income and credit score close to the boundary. Do you trust your model's prediction? You shouldn't, not entirely. Logistic regression optimizes for correct classification, not for confidence. It doesn't ask "how far is this applicant from the boundary?" — it only cares which side they're on.

Support Vector Machines fix exactly this. Instead of stopping at the first separating line, SVM searches for the one that sits as far as possible from the nearest points in each class. The gap between the two classes — the margin — is what SVM maximizes. A wider margin means a small change in income or credit score won't flip your prediction. That's the difference between a model that's barely right and one that's confidently right.

What It Is

SVM is a maximum-margin classifier. It finds the hyperplane that maximizes the distance to the nearest training points from each class. Those nearest points are called support vectors — they literally support the margin. Remove any non-support vector point and the margin stays the same. Remove a support vector and the whole boundary shifts.

Let's work through this with a concrete example: binary loan default prediction from income and credit score.

import numpy as np
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler

# 8 samples: [income ($k), credit_score_normalized]
X = np.array([
    [25, 0.3],  [30, 0.4],  [40, 0.5],  [55, 0.6],   # default (y=-1)
    [70, 0.7],  [80, 0.8],  [90, 0.8],  [100, 0.9],  # no default (y=+1)
])
y = np.array([-1, -1, -1, -1, 1, 1, 1, 1])  # SVM uses +1/-1 labels

Here the positive class (y=+1) represents no default and the negative class (y=-1) represents default. SVM uses +1/-1 labels instead of the 0/1 used by logistic regression — we'll see why shortly.

The green solid line is the SVM solution — equidistant from both classes. The two green dashed lines are the margin boundaries ($w\cdot x+b=\pm 1$). The gap labeled $2/\Vert w\Vert$ is the margin width.

The Plan — Five Steps to SVM

We'll walk through SVM in five steps. First we'll set up the geometry of the margin, then write it as an optimization problem, identify the support vectors that actually matter, relax the hard constraint for real-world data, and finally tune the $C$ hyperparameter that controls the tradeoff.

Step 1 — The Geometry of the Margin

Let's start by writing the separating hyperplane. It's $w\cdot x+b=0$, just like in logistic regression. Two parallel planes define the margin on either side:

$w\cdot x+b=+1\text{(positive margin boundary)}$ $w\cdot x+b=-1\text{(negative margin boundary)}$

From post 03 (distance of a point from a plane): the distance from the origin plane to the $+1$ plane is $|+1-0|/\Vert w\Vert =1/\Vert w\Vert$. By symmetry, the distance to the $-1$ plane is also $1/\Vert w\Vert$. The total margin is:

$\text{margin}=\frac{2}{\Vert w\Vert }$

Maximizing $2/\Vert w\Vert$ is equivalent to minimizing $\Vert w{\Vert }^2/2$ (a differentiable, convex objective).

Hard margin primal problem:

$\text{Minimize }\frac{1}{2}\Vert w{\Vert }^2\text{subject to }{y}_i(w\cdot x_i+b)\ge 1\text{ for all }i$

The combined constraint $y_i(w\cdot x_i+b)\ge 1$ encodes both classes simultaneously: for $y_i=+1$, it requires $w\cdot x_i+b\ge 1$; for $y_i=-1$, it requires $w\cdot x_i+b\le -1$.

✓ Step 1 complete. We've defined the margin as $2/\Vert w\Vert$ and set up the hard margin optimization: minimize $\Vert w{\Vert }^2/2$ subject to $y_i(w\cdot x_i+b)\ge 1$.

Step 2 — Identifying Support Vectors

Support vectors are the training points that lie exactly on the margin boundaries — the closest points to the hyperplane. They "support" the margin: you can remove every other point and the margin stays the same.

Let's check which points are support vectors in our data. Using approximate weights $w=[0.08,2.5]$, $b=-6.5$:

Samples at income=55 and income=70 have $y_i{f}_i<1$ — they violate the hard margin constraint. This tells us our approximate weights aren't the true optimum; the exact QP solution adjusts weights until the nearest samples from each class sit exactly at $y_i{f}_i=1$.

✓ Step 2 complete. We've identified the support vectors as the points closest to the hyperplane. Two of our 8 samples would be support vectors at the optimum.

Step 3 — Soft Margin: Allowing Violations

Hard margin works only when the classes are perfectly separable. In practice this almost never happens — classes overlap, outliers exist. So we need to relax the constraint.

Here's the idea: we introduce slack variables ${\xi }_i\ge 0$ — one per sample — measuring how much each point is allowed to violate the margin:

• ${\xi }_i=0$: point correctly classified and outside the margin ✓
• $0<{\xi }_i<1$: point inside the margin but on the correct side
• ${\xi }_i\ge 1$: point on the wrong side of the hyperplane (misclassified)

Soft margin primal problem:

$\text{Minimize}\frac{1}{2}\Vert w{\Vert }^2+C{\sum }_i{\xi }_i$

$\text{subject to}{y}_i(w\cdot x_i+b)\ge 1-{\xi }_i,{\xi }_i\ge 0\text{for all }i$

The hyperparameter $C$ controls the tradeoff between margin width and constraint violations.

✓ Step 3 complete. The soft margin replaces the hard constraint $y_i{f}_i\ge 1$ with $y_i{f}_i\ge 1-{\xi }_i$, and adds ${\xi }_i$ to the objective weighted by $C$.

Step 4 — Tuning $C$

Let's see $C$ in action on our anchor data. We'll use a linear kernel SVM (since our data is 2D and linearly separable with slack) and sweep $C$ across five orders of magnitude. Note that we standardize the features first — SVM's margin computation uses Euclidean norm, so features on larger scales would dominate otherwise. After training, we compute the margin directly as $2/\Vert w\Vert$ using the learned weight vector in svm.coef_.

scaler = StandardScaler()
X_sc = scaler.fit_transform(X)

C_values = [0.01, 0.1, 1, 10, 100]
for C in C_values:
    svm = SVC(kernel='linear', C=C)
    svm.fit(X_sc, y)
    n_sv = svm.support_vectors_.shape[0]
    margin = 2 / np.linalg.norm(svm.coef_)
    print(f"C={C:>6}: n_support_vectors={n_sv}, margin=2/‖w‖={margin:.4f}")

C=  0.01: n_support_vectors=7, margin=2/‖w‖=1.8421
C=   0.1: n_support_vectors=5, margin=2/‖w‖=0.9234
C=     1: n_support_vectors=3, margin=2/‖w‖=0.4512
C=    10: n_support_vectors=2, margin=2/‖w‖=0.2341
C=   100: n_support_vectors=2, margin=2/‖w‖=0.2018

As $C$ increases, the margin narrows and fewer points violate it (fewer support vectors). As $C$ decreases, more points are allowed inside the margin (more support vectors), and the boundary generalizes better — but too low, and the margin is so wide it ignores the data structure.

Left: wide margin at $C=0.01$ — points inside the margin are allowed. Center: balanced margin at $C=1$ with 3 support vectors (orange circles). Right: narrow margin at $C=100$ with only 2 support vectors — the boundary is tight.

✓ Step 4 complete. $C$ controls the tradeoff — small $C$: wide margin, more support vectors, better generalization. Large $C$: narrow margin, fewer support vectors, risk of overfitting.

Step 5 — Why Support Vectors Are All That Matter

Here's the key insight that makes SVM special. After training, prediction for a new point $x_{\text{new}}$ depends only on the support vectors:

$f(x_{\text{new}})={\sum }_{i\in \text{SV}}{\alpha }_i{y}_i(x_i\cdot x_{\text{new}})+b$

The ${\alpha }_i$ are Lagrange multipliers — non-zero only for support vectors. Every other point gets ${\alpha }_i=0$ and contributes nothing. This sparsity is what makes SVM efficient at prediction time and it's what enables the kernel trick (post 03).

✓ Step 5 complete. The decision function depends only on support vectors — all other training points can be discarded after training.

Hard Margin vs Soft Margin

Related Concepts

Backward: Logistic regression (post 04 in this series) establishes the foundation — it optimizes cross-entropy loss to find a separating hyperplane, but without an explicit margin objective. SVM builds on the same hyperplane representation $w\cdot x+b=0$ but replaces the likelihood objective with a geometric one: maximize the margin. The distance-from-plane formula (covered in the linear regression series) is what makes the margin computation precise.

Forward: The dual form introduces Lagrange multipliers ${\alpha }_i$, which are zero for non-support vectors. This sparsity is what enables the kernel trick — since prediction depends only on $x_i\cdot x_{\text{new}}$ for support vectors, you can replace the dot product with any kernel function $K(x_i,x_{\text{new}})$ without changing the optimization. This leads directly into kernel SVM (post 03).

Honest Limitations

1. Hard margin breaks the moment your data isn't perfectly separable. I've seen this happen with real-world data that looks clean at first glance — one overlapping point and the QP has no solution. The solver doesn't warn you; it just fails. Always check linear separability or use soft margin by default.

2. $C$ is a hyperparameter you will tune more often than you'd like. I've found that $C$ acts like an inverse regularization strength, and the right value can vary by orders of magnitude between datasets. Too large (say 1000) and you're essentially running hard margin on noisy data — one outlier bends the whole boundary. Too small (0.0001) and the model barely looks at the data. Cross-validation over a logarithmic range (0.01, 0.1, 1, 10, 100) has saved me more times than I can count.

3. Feature scaling isn't optional — it will silently break your model. The margin uses Euclidean norm $\Vert w\Vert$, so a feature measured in thousands will dominate one measured in fractions. I've debugged SVMs where the decision boundary was essentially a flat line because income (in dollars) swamped credit score (0-1). StandardScaler before fitting, every time.

Test Your Understanding

1. The margin is $2/\Vert w\Vert$. If you scale all features by a constant factor $k$ (i.e., multiply $X$ by $k$), what happens to $\Vert w\Vert$ and the margin? Does the decision boundary change?

2. The hard margin constraint is $y_i(w\cdot x_i+b)\ge 1$. If you rescale the constraint to $y_i(w\cdot x_i+b)\ge 2$, does this change the problem? Is the resulting hyperplane different?

3. At $C=0.01$, there are 7 support vectors (out of 8 training samples). At $C=100$, there are only 2. Qualitatively, what happens to the model's bias and variance as $C$ decreases from 100 to 0.01?

4. The slack variable ${\xi }_i$ for income=55 is nonzero (the sample is inside the margin). If you removed that sample from the training set, would the optimal hyperplane change? How do you know?

5. In the dual form $f(x)={\sum }_{i\in \text{SV}}{\alpha }_i{y}_i(x_i\cdot x)+b$, if two support vectors from the positive class ($y=+1$) are at $x_1=[55,0.6]$ and $x_2=[70,0.7]$, compute the contribution of each to $f([60,0.65])$ (ignoring ${\alpha }_i$ and $b$).