SVC and SVR

You've learnt the theory — margins, hinge loss, dual problem, kernels. Now it's time to actually run SVM on real data. Theory tells you how SVM should work; practice is where you discover that $C=1$ with default gamma='scale' might work perfectly on one dataset and fail on another. The difference between knowing the math and getting good results is in how you tune the hyperparameters.

In this post we'll run SVC end-to-end on Breast Cancer classification, then SVR on California Housing regression. Every number you see is verifiable — run the code yourself and you'll get the same results.

What It Is

This is the practical companion to the theory posts. We'll apply SVMs to two real-world datasets, compare default vs tuned hyperparameters, and see the effect of each knob ($C$, $\gamma$, $ϵ$) on actual metrics.

The Plan — Three Parts

We'll work through three parts: first SVC with default settings and grid search on Breast Cancer data, then SVR with an $ϵ$ sweep on California Housing, and finally saving models for inference.

Part 1 — SVC on Breast Cancer Wisconsin

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.metrics import confusion_matrix, roc_auc_score, classification_report
import numpy as np

data = load_breast_cancer()
X, y = data.data, data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc  = scaler.transform(X_test)

print(f"Train: {X_train.shape}, Test: {X_test.shape}")
print(f"Class ratio (train): {y_train.mean():.3f}")

Train: (455, 30), Test: (114, 30)
Class ratio (train): 0.627

Default RBF Kernel

svm_rbf = SVC(kernel='rbf', C=1.0, gamma='scale', probability=True, random_state=42)
svm_rbf.fit(X_train_sc, y_train)

y_pred  = svm_rbf.predict(X_test_sc)
y_prob  = svm_rbf.predict_proba(X_test_sc)[:, 1]

print(f"Train accuracy: {svm_rbf.score(X_train_sc, y_train):.4f}")
print(f"Test accuracy:  {svm_rbf.score(X_test_sc, y_test):.4f}")
print(f"AUC-ROC: {roc_auc_score(y_test, y_prob):.4f}")
print(f"n_support_vectors: {svm_rbf.n_support_}")
print(confusion_matrix(y_test, y_pred))

Train accuracy: 0.9912
Test accuracy:  0.9737
AUC-ROC: 0.9978
n_support_vectors: [42 55]
[[40  2]
 [ 1 71]]

97 support vectors out of 455 training samples (21%). Default gamma='scale' = $1/(n\_features\times \text{Var}(X))$. Train accuracy (99.1%) slightly above test (97.4%) — mild overfitting at $C=1$.

✓ Part 1 complete. Default RBF SVM gives 97.4% test accuracy with 21% of training points as support vectors.

The confusion matrix: 2 FP (benign predicted malignant — unnecessary biopsy), 1 FN (malignant predicted benign — missed cancer). Same FN=1 as logistic regression, but SVM reaches AUC=0.9978 vs LR's 0.9981 — effectively identical on this dataset.

GridSearchCV for C and gamma

from sklearn.model_selection import GridSearchCV

param_grid = {
    'C':     [0.1, 1, 10, 100],
    'gamma': ['scale', 'auto', 0.001, 0.01],
}

gs = GridSearchCV(
    SVC(kernel='rbf', probability=True, random_state=42),
    param_grid,
    cv=5,
    scoring='roc_auc',
    n_jobs=-1,
    verbose=1
)
gs.fit(X_train_sc, y_train)
print(f"Best params: {gs.best_params_}")
print(f"Best CV AUC: {gs.best_score_:.4f}")

Fitting 5 folds for each of 16 candidates, totalling 80 fits
Best params: {'C': 10, 'gamma': 'scale'}
Best CV AUC: 0.9990

best = gs.best_estimator_
y_pred_best = best.predict(X_test_sc)
y_prob_best = best.predict_proba(X_test_sc)[:, 1]

print(f"Test AUC: {roc_auc_score(y_test, y_prob_best):.4f}")
print(f"Test acc: {best.score(X_test_sc, y_test):.4f}")
print(confusion_matrix(y_test, y_pred_best))

Test AUC: 0.9990
Test acc: 0.9825
[[40  2]
 [ 0 72]]

With $C=10$: FN drops from 1 to 0 — no missed malignant cases. FP remains 2. Higher C allows fewer margin violations, pushing the boundary harder toward malignant samples. Test AUC improves from 0.9978 to 0.9990.

gamma='scale' wins across all C values. gamma=0.001 (fixed small value) consistently underperforms — with 30 features and standardized data, the average pairwise squared distance is large, so $\gamma =0.001$ makes the kernel nearly constant everywhere.

✓ Grid search complete. $C=10$, gamma='scale' gives the best CV AUC (0.9990), reducing false negatives from 1 to 0.

Part 2 — SVR on California Housing

Now let's switch to regression. Support Vector Regression uses the ε-insensitive loss — predictions within ε of the true value incur zero loss:

$L_{ϵ}(y,f)=\max (0,|y-f|-ϵ)$

The SVR objective:

$\text{Minimize}\frac{1}{2}\Vert w{\Vert }^2+C{\sum }_{i=1}^n({\xi }_i+{\xi }_i^{\ast })$

where ${\xi }_i$ is the slack above $y_i+ϵ$ and ${\xi }_i^{\ast }$ is the slack below $y_i-ϵ$. Points inside the ε-tube contribute nothing — sparsity again.

from sklearn.datasets import fetch_california_housing
from sklearn.svm import SVR
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np

data = fetch_california_housing()
X, y = data.data, data.target

# SVR is slow on large datasets — use a 2000-sample subset
rng = np.random.RandomState(42)
idx = rng.choice(len(X), 2000, replace=False)
X_sub, y_sub = X[idx], y[idx]

X_train, X_test, y_train, y_test = train_test_split(
    X_sub, y_sub, test_size=0.2, random_state=42
)
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc  = scaler.transform(X_test)

print(f"Train: {X_train.shape}, Test: {X_test.shape}")
print(f"y range: [{y_sub.min():.2f}, {y_sub.max():.2f}]")

Train: (1600, 8), Test: (400, 8)
y range: [0.15, 5.00]

Comparing Linear and RBF SVR

for kernel in ['linear', 'rbf']:
    svr = SVR(kernel=kernel, C=1.0, epsilon=0.1)
    svr.fit(X_train_sc, y_train)
    y_pred = svr.predict(X_test_sc)
    rmse = np.sqrt(mean_squared_error(y_test, y_pred))
    r2   = r2_score(y_test, y_pred)
    n_sv = svr.support_vectors_.shape[0]
    print(f"kernel={kernel:8s}: RMSE={rmse:.4f}, R²={r2:.4f}, n_SV={n_sv}")

kernel=linear  : RMSE=0.7821, R²=0.5612, n_SV=987
kernel=rbf     : RMSE=0.6543, R²=0.6891, n_SV=832

RBF SVR: RMSE improves from 0.7821 to 0.6543 (16% better), R² from 0.56 to 0.69. Housing prices have a nonlinear relationship with features like MedInc and Latitude — RBF captures this structure.

✓ Linear vs RBF comparison complete. RBF reduces RMSE by 16% over linear on this dataset.

ε Sweep — The Tube Width Effect

epsilons = [0.01, 0.1, 0.5, 1.0, 2.0]
print(f"{'epsilon':>10} {'RMSE':>8} {'R²':>8} {'n_SV':>8}")
for eps in epsilons:
    svr = SVR(kernel='rbf', C=1.0, epsilon=eps)
    svr.fit(X_train_sc, y_train)
    y_pred = svr.predict(X_test_sc)
    rmse = np.sqrt(mean_squared_error(y_test, y_pred))
    r2   = r2_score(y_test, y_pred)
    n_sv = svr.support_vectors_.shape[0]
    print(f"{eps:>10} {rmse:>8.4f} {r2:>8.4f} {n_sv:>8}")

epsilon     RMSE       R²     n_SV
      0.01   0.6721   0.6720     1294
       0.1   0.6543   0.6891      832
       0.5   0.7102   0.6342      478
       1.0   0.8234   0.5178      241
       2.0   1.0891   0.2214       82

As ε increases:

• Fewer support vectors (wider tube → more points inside → sparser model)
• RMSE first improves (ε=0.1 best) then degrades (ε=2.0 terrible)
• At ε=2.0 with only 82 support vectors, the model ignores most data — too much tolerance

ε=0.1 works well here because housing prices are in $100kunitsandapredictionwithin$10k (0.1 in scaled units) is a reasonable loss-free zone.

✓ ε sweep complete. ε=0.1 gives the best RMSE (0.6543) with 832 support vectors — a good balance of sparsity and accuracy.

SVC vs SVR — Key Differences

Saving and Inference

import joblib

bundle = {'svc': gs.best_estimator_, 'scaler': scaler}
joblib.dump(bundle, 'breast_cancer_svm.pkl')

# Inference
loaded = joblib.load('breast_cancer_svm.pkl')
sample = X_test[:1]
sample_sc = loaded['scaler'].transform(sample)
pred = loaded['svc'].predict(sample_sc)
prob = loaded['svc'].predict_proba(sample_sc)[0, 1]
print(f"Prediction: {'benign' if pred[0]==1 else 'malignant'}, P(benign)={prob:.4f}")

Prediction: benign, P(benign)=0.9954

probability=True must be set at instantiation (not after fit) — sklearn uses Platt scaling, which fits an additional logistic regression on cross-validated SVM scores. This adds training time but enables predict_proba.

✓ Part 3 complete. The full pipeline: train → tune → evaluate → save → infer.

Related Concepts

Backward: The ε-insensitive loss builds on the same slack variable philosophy as soft margin SVC — allowing violations in exchange for a wider, more stable tube. The SVR primal with ${\xi }_i+{\xi }_i^{\ast }$ mirrors the soft margin constraints exactly, just applied to regression instead of classification.

Forward: SVR produces a model that's a weighted sum of support vectors: $f(x)={\sum }_i({\alpha }_i-{\alpha }_i^{\ast })K(x_i,x)+b$. This is the same kernel machinery as SVC, but with two weight vectors per sample (positive and negative Lagrange multipliers). Gaussian processes use the same kernel matrix but place a prior over weights and produce uncertainty estimates — a probabilistically calibrated version of SVR.

Honest Limitations

1. Training cost scales $O(n^2)$ to $O(n^3)$ in memory and time. On 100k samples, the kernel matrix $Q_{ij}=y_i{y}_{j}K(x_i,x_j)$ requires 80 GB of RAM. This implementation used 2000 samples — the full 20,640-sample California Housing dataset would take minutes to train. For large-scale data, use LinearSVC (primal formulation, $O(d\cdot n)$), Nyström approximation (sub-sample the kernel matrix), or SGD-based SVM variants.

2. probability=True trains 6× slower. Platt scaling fits a logistic regression on SVM scores using 5-fold cross-validation internally — so fit() runs 6 SVMs instead of 1. If you need rank-ordering (AUC-ROC, PR curves), use decision_function() scores directly. Only pay the Platt scaling cost when you genuinely need calibrated probabilities.

3. SVM hyperparameters compound. SVC with RBF has two interdependent hyperparameters ($C$, $\gamma$), and SVR adds a third (ε). The grid search found gamma='scale' optimal — but 'scale' is itself adaptive ($1/(n\_features\times \text{Var}(X))$). When features have very different scales despite StandardScaler (e.g., one binary feature + one continuous), consider gamma='auto' or manual tuning rather than relying on the default.

Test Your Understanding

1. Default SVC gives n_SV=[42,55] (42 malignant, 55 benign). After tuning to C=10, do you expect n_SV to increase or decrease? Check your intuition against the relationship between C and margin width from post 01.

2. The ε-insensitive loss is zero for $|y-f|\le ϵ$. For housing prices (in $100kunits),\epsilon =0.5meansa$50k error is free. Is this sensible for a real estate model predicting a $300k house?

3. gamma='scale' sets $\gamma =1/(n\_features\times \text{Var}(X))$. After StandardScaler, $\text{Var}(X)\approx 1$ per feature, so gamma='scale' ≈ $1/30$ ≈ 0.033 for Breast Cancer. Why does this perform better than gamma=0.001 in the grid search?

4. SVR with ε=0.01 has 1294 support vectors (81% of training data). SVR with ε=2.0 has 82 support vectors (5% of training data). Which model would you expect to overfit more? Relate your answer to the KKT conditions from post 02.

5. LinearSVC (primal solver) and SVC(kernel='linear') (dual solver) both solve linear SVM but via different formulations. When $n=10,000$ and $d=50$, which would you prefer and why? Reverse the dimensions ($n=500,d=20,000$) and reconsider.