AdaBoost: Implementation and Hyperparameter Tuning

The previous post traced AdaBoost by hand. This post implements it with sklearn, exposes the internal learner weights, and sweeps hyperparameters to show what breaks AdaBoost and what makes it work.

Anchor: Breast Cancer Wisconsin — 569 samples, 30 features, binary classification.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

data = load_breast_cancer()
X, y = data.data, data.target  # 569 samples, 30 features
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Train: {X_train.shape}, Test: {X_test.shape}")

AdaBoostClassifier: Default Run

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier

ada = AdaBoostClassifier(
    estimator=DecisionTreeClassifier(max_depth=1),
    n_estimators=50,
    learning_rate=1.0,
    algorithm='SAMME.R',
    random_state=42
)
ada.fit(X_train, y_train)
print(f"Train accuracy: {ada.score(X_train, y_train):.4f}")
print(f"Test accuracy:  {ada.score(X_test, y_test):.4f}")

50 depth-1 stumps combining via SAMME.R achieve 96.5% test accuracy with training error = 0.

SAMME vs SAMME.R

sklearn offers two algorithms for AdaBoost:

SAMME.R (R=Real) uses probability estimates rather than hard votes at each round, giving more gradient information and converging faster.

Inspecting Learner Weights

Each stump in the ensemble has a weight (how trusted it is) and an error:

weights = ada.estimator_weights_
errors  = ada.estimator_errors_

print(f"Number of stumps: {len(weights)}")
print(f"\nFirst 5 stump weights (α): {weights[:5].round(4)}")
print(f"First 5 stump errors (ε): {errors[:5].round(4)}")

print(f"\nMin error stump: ε={errors.min():.4f} → α={weights.max():.4f}")
print(f"Max error stump: ε={errors.max():.4f} → α={weights.min():.4f}")

Verify: $\alpha =\frac{1}{2}\ln \frac{1-\epsilon }{\epsilon }$ for the best stump:

$\alpha =\frac{1}{2}\ln \frac{1-0.0523}{0.0523}=\frac{1}{2}\ln (17.1)=\frac{1}{2}\times 2.839=1.420$

Close to the reported 1.475 — slight difference because SAMME.R uses a continuous update rather than the discrete SAMME formula.

Staged Prediction: How Accuracy Evolves

staged_predict returns predictions at each intermediate round — a free performance curve:

from sklearn.metrics import accuracy_score

train_accs = []
test_accs  = []

for y_pred_train, y_pred_test in zip(
    ada.staged_predict(X_train),
    ada.staged_predict(X_test)
):
    train_accs.append(accuracy_score(y_train, y_pred_train))
    test_accs.append(accuracy_score(y_test,  y_pred_test))

for r in [1, 5, 10, 20, 30, 50]:
    print(f"Round {r:3d}: Train={train_accs[r-1]:.4f}, Test={test_accs[r-1]:.4f}")

Training accuracy reaches 1.0 at round 30. Test accuracy plateaus at round 20 (0.9649) and stays flat. No degradation at n=50 on this clean dataset — AdaBoost is not overfitting here.

Hyperparameter Sweep: n_estimators

for n in [10, 20, 30, 50, 100, 200, 300, 500]:
    ada_n = AdaBoostClassifier(
        estimator=DecisionTreeClassifier(max_depth=1),
        n_estimators=n, learning_rate=1.0, random_state=42
    )
    ada_n.fit(X_train, y_train)
    print(f"n={n:4d}: Test={ada_n.score(X_test, y_test):.4f}")

AdaBoost is more robust to overestimating n than Gradient Boosting. On clean datasets, adding extra rounds after convergence often doesn't hurt much. On noisy data, it can.

Hyperparameter Sweep: learning_rate

learning_rate (ν) scales each stump's contribution: $F(x)={\sum }_t\nu {\alpha }_t{h}_t(x)$. Lower ν = smaller steps = more regularization, but needs more estimators.

for lr in [0.01, 0.1, 0.5, 1.0, 2.0]:
    ada_lr = AdaBoostClassifier(
        estimator=DecisionTreeClassifier(max_depth=1),
        n_estimators=200, learning_rate=lr, random_state=42
    )
    ada_lr.fit(X_train, y_train)
    print(f"lr={lr:.2f}: Test={ada_lr.score(X_test, y_test):.4f}")

The tradeoff: lr=0.1 + n=500 ≈ lr=1.0 + n=50 — more rounds with smaller steps often performs similarly. The classic guideline: lower lr (0.01–0.1) with high n gives better generalization on noisy data.

Hyperparameter Sweep: Base Learner Depth

AdaBoost doesn't require depth-1 stumps. Deeper trees are valid base learners:

for depth in [1, 2, 3, 5, None]:
    ada_d = AdaBoostClassifier(
        estimator=DecisionTreeClassifier(max_depth=depth),
        n_estimators=50, learning_rate=1.0, random_state=42
    )
    ada_d.fit(X_train, y_train)
    print(f"depth={str(depth):4s}: Train={ada_d.score(X_train, y_train):.4f}, Test={ada_d.score(X_test, y_test):.4f}")

Deeper trees → worse test accuracy. AdaBoost needs weak base learners. If each tree already captures complex decision boundaries, boosting amplifies the memorization rather than correcting bias. The theory: AdaBoost reduces bias by combining many high-bias models; giving it low-bias models breaks the assumption.

Full Evaluation

from sklearn.metrics import classification_report, roc_auc_score, confusion_matrix

best_ada = AdaBoostClassifier(
    estimator=DecisionTreeClassifier(max_depth=1),
    n_estimators=50, learning_rate=1.0, algorithm='SAMME.R', random_state=42
)
best_ada.fit(X_train, y_train)
y_pred = best_ada.predict(X_test)
y_prob = best_ada.predict_proba(X_test)[:, 1]

print(classification_report(y_test, y_pred, target_names=['Malignant', 'Benign']))
print(f"AUC-ROC: {roc_auc_score(y_test, y_prob):.4f}")
print(f"\nConfusion Matrix:")
print(confusion_matrix(y_test, y_pred))

FP=1: one malignant tumor predicted as benign (dangerous miss). FN=3: three benign tumors predicted as malignant (unnecessary follow-up). AUC=0.9923 — the model's probability rankings are nearly perfect.

AdaBoost vs Random Forest

Test Your Understanding

1. The best stump in the ensemble has ε=0.0523, α=1.475. Verify using the SAMME formula: $\alpha =\frac{1}{2}\ln \frac{1-\epsilon }{\epsilon }$. You get ≈1.420, not 1.475. The discrepancy exists because sklearn uses SAMME.R (probability-based update). In SAMME.R, what replaces the hard class label in the weight update — and why does this produce a different α?

2. The n_estimators sweep shows mild degradation at n=300+ (0.965 → 0.956). Yet the staged_predict curve showed test accuracy plateauing at round 20 and staying flat through round 50. If accuracy was flat at round 50, why does adding 250 more rounds (300 total) cause degradation?

3. At learning_rate=0.01 with n=200 estimators: test accuracy=0.9386 — same as n=10 with lr=1.0. If you increased n to 2000 at lr=0.01, would the test accuracy eventually match lr=1.0 at n=50? What constraint prevents you from always using lr=0.01 + very high n in practice?

4. The depth sweep shows that depth=None (full trees) achieves test=0.921, worse than depth=1 (0.965). Both achieve train=1.0. A single depth=None tree on this dataset achieves test≈0.92 without boosting. Why doesn't boosting 50 full trees improve on a single full tree, even though boosting 50 stumps improves dramatically over a single stump?

5. Confusion matrix: FP=1, FN=3. In a cancer screening context, FP (predicting malignant when benign) means unnecessary biopsy; FN (predicting benign when malignant) means missed cancer. To reduce FN from 3 to 1 (accept more FP), should you raise or lower the decision threshold? What code change would you make to implement this?