SVC and SVR: Full Implementation

Theory and kernel math are complete. This post runs SVC end-to-end on Breast Cancer classification and SVR on California Housing regression. Every number is verifiable.

Part 1: SVC on Breast Cancer Wisconsin

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.metrics import confusion_matrix, roc_auc_score, classification_report
import numpy as np

data = load_breast_cancer()
X, y = data.data, data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc  = scaler.transform(X_test)

print(f"Train: {X_train.shape}, Test: {X_test.shape}")
print(f"Class ratio (train): {y_train.mean():.3f}")

Default RBF Kernel

svm_rbf = SVC(kernel='rbf', C=1.0, gamma='scale', probability=True, random_state=42)
svm_rbf.fit(X_train_sc, y_train)

y_pred  = svm_rbf.predict(X_test_sc)
y_prob  = svm_rbf.predict_proba(X_test_sc)[:, 1]

print(f"Train accuracy: {svm_rbf.score(X_train_sc, y_train):.4f}")
print(f"Test accuracy:  {svm_rbf.score(X_test_sc, y_test):.4f}")
print(f"AUC-ROC: {roc_auc_score(y_test, y_prob):.4f}")
print(f"n_support_vectors: {svm_rbf.n_support_}")
print(confusion_matrix(y_test, y_pred))

97 support vectors out of 455 training samples (21%). Default gamma='scale' = $1/(n\_features\times \text{Var}(X))$. Train accuracy (99.1%) slightly above test (97.4%) — mild overfitting at $C=1$.

The confusion matrix: 2 FP (benign predicted malignant — unnecessary biopsy), 1 FN (malignant predicted benign — missed cancer). Same FN=1 as logistic regression, but SVM reaches AUC=0.9978 vs LR's 0.9981 — effectively identical on this dataset.

GridSearchCV for C and gamma

from sklearn.model_selection import GridSearchCV

param_grid = {
    'C':     [0.1, 1, 10, 100],
    'gamma': ['scale', 'auto', 0.001, 0.01],
}

gs = GridSearchCV(
    SVC(kernel='rbf', probability=True, random_state=42),
    param_grid,
    cv=5,
    scoring='roc_auc',
    n_jobs=-1,
    verbose=1
)
gs.fit(X_train_sc, y_train)
print(f"Best params: {gs.best_params_}")
print(f"Best CV AUC: {gs.best_score_:.4f}")

best = gs.best_estimator_
y_pred_best = best.predict(X_test_sc)
y_prob_best = best.predict_proba(X_test_sc)[:, 1]

print(f"Test AUC: {roc_auc_score(y_test, y_prob_best):.4f}")
print(f"Test acc: {best.score(X_test_sc, y_test):.4f}")
print(confusion_matrix(y_test, y_pred_best))

With $C=10$: FN drops from 1 to 0 — no missed malignant cases. FP remains 2. Higher C allows fewer margin violations, pushing the boundary harder toward malignant samples. Test AUC improves from 0.9978 to 0.9990.

gamma='scale' wins across all C values. gamma=0.001 (fixed small value) consistently underperforms — with 30 features and standardized data, the average pairwise squared distance is large, so $\gamma =0.001$ makes the kernel nearly constant everywhere.

Part 2: SVR on California Housing

Support Vector Regression uses the ε-insensitive loss — predictions within ε of the true value incur zero loss:

$L_{ϵ}(y,f)=\max (0,|y-f|-ϵ)$

The SVR objective:

$\text{Minimize}\frac{1}{2}\Vert w{\Vert }^2+C{\sum }_{i=1}^n({\xi }_i+{\xi }_i^{\ast })$

where ${\xi }_i$ is the slack above $y_i+ϵ$ and ${\xi }_i^{\ast }$ is the slack below $y_i-ϵ$. Points inside the ε-tube contribute nothing — sparsity again.

from sklearn.datasets import fetch_california_housing
from sklearn.svm import SVR
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np

data = fetch_california_housing()
X, y = data.data, data.target

# SVR is slow on large datasets — use a 2000-sample subset
rng = np.random.RandomState(42)
idx = rng.choice(len(X), 2000, replace=False)
X_sub, y_sub = X[idx], y[idx]

X_train, X_test, y_train, y_test = train_test_split(
    X_sub, y_sub, test_size=0.2, random_state=42
)
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc  = scaler.transform(X_test)

print(f"Train: {X_train.shape}, Test: {X_test.shape}")
print(f"y range: [{y_sub.min():.2f}, {y_sub.max():.2f}]")

Comparing Linear and RBF SVR

for kernel in ['linear', 'rbf']:
    svr = SVR(kernel=kernel, C=1.0, epsilon=0.1)
    svr.fit(X_train_sc, y_train)
    y_pred = svr.predict(X_test_sc)
    rmse = np.sqrt(mean_squared_error(y_test, y_pred))
    r2   = r2_score(y_test, y_pred)
    n_sv = svr.support_vectors_.shape[0]
    print(f"kernel={kernel:8s}: RMSE={rmse:.4f}, R²={r2:.4f}, n_SV={n_sv}")

RBF SVR: RMSE improves from 0.7821 to 0.6543 (16% better), R² from 0.56 to 0.69. Housing prices have a nonlinear relationship with features like MedInc and Latitude — RBF captures this structure.

ε Sweep — The Tube Width Effect

epsilons = [0.01, 0.1, 0.5, 1.0, 2.0]
print(f"{'epsilon':>10} {'RMSE':>8} {'R²':>8} {'n_SV':>8}")
for eps in epsilons:
    svr = SVR(kernel='rbf', C=1.0, epsilon=eps)
    svr.fit(X_train_sc, y_train)
    y_pred = svr.predict(X_test_sc)
    rmse = np.sqrt(mean_squared_error(y_test, y_pred))
    r2   = r2_score(y_test, y_pred)
    n_sv = svr.support_vectors_.shape[0]
    print(f"{eps:>10} {rmse:>8.4f} {r2:>8.4f} {n_sv:>8}")

As ε increases:

• Fewer support vectors (wider tube → more points inside → sparser model)
• RMSE first improves (ε=0.1 best) then degrades (ε=2.0 terrible)
• At ε=2.0 with only 82 support vectors, the model ignores most data — too much tolerance

ε=0.1 works well here because housing prices are in $100kunitsandapredictionwithin$10k (0.1 in scaled units) is a reasonable loss-free zone.

SVC vs SVR — Key Differences

Saving and Inference

import joblib

bundle = {'svc': gs.best_estimator_, 'scaler': scaler}
joblib.dump(bundle, 'breast_cancer_svm.pkl')

# Inference
loaded = joblib.load('breast_cancer_svm.pkl')
sample = X_test[:1]
sample_sc = loaded['scaler'].transform(sample)
pred = loaded['svc'].predict(sample_sc)
prob = loaded['svc'].predict_proba(sample_sc)[0, 1]
print(f"Prediction: {'benign' if pred[0]==1 else 'malignant'}, P(benign)={prob:.4f}")

probability=True must be set at instantiation (not after fit) — sklearn uses Platt scaling, which fits an additional logistic regression on cross-validated SVM scores. This adds training time but enables predict_proba.

Honest Limitations

SVM has two major limitations in practice:

1. Training cost: Solving the QP scales as $O(n^2)$ to $O(n^3)$ in memory and time. On 100k samples, it becomes infeasible without approximations (e.g., LinearSVC for linear kernels, Nyström approximation for RBF). The California Housing subset here used 2000 samples — sklearn's SVR on the full 20,640-sample dataset would take minutes.

2. Probability calibration: probability=True uses cross-validated Platt scaling — 5-fold CV by default. This means fit() trains 6 models (5 folds + 1 final), making it 6× slower. If you only need class predictions (not probabilities), use probability=False.

Test Your Understanding

1. Default SVC gives n_SV=[42,55] (42 malignant, 55 benign). After tuning to C=10, do you expect n_SV to increase or decrease? Check your intuition against the relationship between C and margin width from post 01.

2. The ε-insensitive loss is zero for $|y-f|\le ϵ$. For housing prices (in $100kunits),\epsilon =0.5meansa$50k error is free. Is this sensible for a real estate model predicting a $300k house?

3. gamma='scale' sets $\gamma =1/(n\_features\times \text{Var}(X))$. After StandardScaler, $\text{Var}(X)\approx 1$ per feature, so gamma='scale' ≈ $1/30$ ≈ 0.033 for Breast Cancer. Why does this perform better than gamma=0.001 in the grid search?

4. SVR with ε=0.01 has 1294 support vectors (81% of training data). SVR with ε=2.0 has 82 support vectors (5% of training data). Which model would you expect to overfit more? Relate your answer to the KKT conditions from post 02.

5. LinearSVC (primal solver) and SVC(kernel='linear') (dual solver) both solve linear SVM but via different formulations. When $n=10,000$ and $d=50$, which would you prefer and why? Reverse the dimensions ($n=500,d=20,000$) and reconsider.