Bagging and Boosting: Ensemble Intuition

You're building a loan approval system. A single decision stump catches 7 of 8 borderline cases, but the one it misses — a borrower whose income sits just above the cutoff — costs the bank real money. Train a logistic regression instead and it bends the decision boundary differently. Some cases switch from approved to denied, but some are still wrong.

Change one row of training data and the boundary shifts. Add a new feature and the sweet spot moves. This fragility isn't unique to any particular model — it's a consequence of learning from finite data.

That's why you reach for ensemble methods. Instead of chasing a single perfect model, you build many imperfect ones and combine them so their individual errors cancel. The two main approaches, bagging and boosting, target different failure modes: bagging averages many high-variance models to smooth out random errors; boosting corrects high-bias models by forcing each new learner to focus on the mistakes of the previous.

What Are Ensemble Methods?

An ensemble combines multiple models into a single prediction. The core insight: if each model makes independent errors, combining them cancels the errors without canceling the signal. Bagging (Bootstrap AGGregating) trains models in parallel on different bootstrap samples of the data, then averages their predictions — which reduces variance. Boosting trains models sequentially, each one focused on the samples the previous models got wrong — which reduces bias.

Anchor dataset: 8-sample loan default dataset.

import numpy as np
import pandas as pd

# 8 samples: [income_$k, credit_score]
X = np.array([
    [25, 580], [32, 610], [45, 650], [60, 680],
    [70, 710], [80, 730], [90, 750], [110, 780]
])
y = np.array([1, 1, 1, 0, 0, 0, 0, 0])  # 1=default
# Class boundary near income≈55k

The Plan — 8 Steps to Bagging and Boosting

We'll work through ensemble methods in two passes. First, bagging in three steps: bootstrap the data, train a model per bootstrap, then aggregate by majority vote. Then, boosting in five steps: initialize weights, train a weak learner, update weights, train the next learner, and combine the weighted predictions. Every step traces through the same 8-sample loan dataset so you can follow the numbers change.

Bagging — Bootstrap AGGregating

Bagging creates diversity by training each model on a different random sample of the data. Each sample is drawn with replacement (bootstrap), so each model sees a slightly different view of the training set.

Step 1: Bootstrap Sampling

Draw n=8 samples WITH REPLACEMENT from the 8 training samples. Each bootstrap has ~63.2% unique samples; the rest are duplicates. The ~36.8% of samples not drawn are called out-of-bag (OOB) samples.

Each bootstrap produces a training set where some samples appear 2–3 times and others are absent. The duplicated samples get more influence over the model trained on that bootstrap.

✓ Step 1 complete. Generated 3 bootstrap samples with replacement. Each has 8 indices, ~63% unique. OOB samples identified for free validation.

Step 2: Train One Model Per Bootstrap

Train a decision stump (depth=1) on each bootstrap. Bootstrap 1 has [25,580] twice and [45,650] twice, so these low-income defaulters dominate the weighted split. Each tree learns a slightly different boundary because it saw a different data distribution.

✓ Step 2 complete. Three stumps trained on different bootstraps. Each finds a slightly different split threshold.

Step 3: Aggregate Predictions

For x_new = [55k income, 670 credit score]:

Two trees predict default, one predicts no_default. Majority vote → default. The ensemble overrides the outlier tree.

✓ Step 3 complete. For x_new=[55,670], majority vote returns default (1). Individual tree disagreement averaged out.

OOB Error Estimation

Each sample is OOB for some trees but not others. Use those trees to estimate its class — no separate validation set needed:

• Sample 3 (income=60k, y=0): OOB for Bootstrap 1 only → Tree 1 predicted default (1) → wrong
• Sample 7 (income=110k, y=0): OOB for Bootstrap 1 only → Tree 1 predicted no_default (0) → correct

OOB accuracy from these 2 samples: 50%. This increases substantially with more trees (typically 50–200 trees give reliable OOB estimates). The OOB error approximates the leave-one-out cross-validation error — a free validation score without a dedicated validation split.

Why Bagging Reduces Variance

A single deep decision tree has high variance — small changes to training data produce very different trees. Bootstrap samples create T different "versions" of the training data → T different trees. When these trees disagree (as Tree 3 did above), the majority vote suppresses the outlier. Mathematically:

$\text{Var}\left(\frac{1}{T}{\sum }_{t=1}^T{h}_t(x)\right)=\rho \cdot {\sigma }^2+\frac{1-\rho }{T}\cdot {\sigma }^2$

Where $\rho$ is the average pairwise correlation between trees, ${\sigma }^2$ is the per-tree variance, and $T$ is the number of trees. As $T\to \infty$: $\text{Var}\to \rho {\sigma }^2$. Trees with lower correlation give bigger variance reductions — this is why Random Forest adds feature subsampling on top of bagging.

Boosting — Sequential Error Correction

Boosting is architecturally opposite to bagging. Instead of parallel independent models, boosting trains models sequentially — each model focuses on the errors the previous models made.

Step 1: Initialize Weights

All 8 samples start with equal weight: $w_i=1/n=1/8=0.125$.

✓ Step 1 complete. Sample weights initialized uniformly at 0.125 each.

Step 2: Train Weak Learner 1

A decision stump (depth=1, income ≤ 55k) — the same stump we started with. It correctly classifies 7 of 8 samples. Sample 3 (income=60k, y=0) is predicted as default but is actually no_default.

Weighted error rate: ${\epsilon }_1=w_3=0.125$ (only sample 3 is wrong, weight = 0.125).

Learner weight: ${\alpha }_1=\frac{1}{2}\ln \left(\frac{1-{\epsilon }_1}{{\epsilon }_1}\right)=\frac{1}{2}\ln \left(\frac{0.875}{0.125}\right)=\frac{1}{2}\ln (7)=0.973$

High ${\alpha }_1$ (close to 1) means this stump is trusted heavily — it was nearly perfect.

✓ Step 2 complete. Stump 1 splits at income ≤ 55k, ε₁=0.125, α₁=0.973. Sample 3 is the lone mistake.

Step 3: Update Sample Weights

Upweight the misclassified sample (sample 3) so the next model is forced to get it right:

• Misclassified (sample 3): $w_3\leftarrow 0.125\times e^{0.973}=0.125\times 2.645=0.331$
• Correct (samples 0–2, 4–7): $w_i\leftarrow 0.125\times e^{-0.973}=0.125\times 0.378=0.047$

Normalize (sum = $7\times 0.047+1\times 0.331=0.329+0.331=0.660$):

Sample 3 now carries 50.1% of the total weight. Any stump that ignores sample 3 will have at least 50% weighted error — worse than random guessing.

✓ Step 3 complete. Sample 3 upweighted from 0.125→0.501 (50% of total weight). Correct samples downweighted to 0.071 each.

Step 4: Train Weak Learner 2

The new stump must focus on classifying sample 3 correctly. The best split considering the new weights might be: credit_score ≤ 695 → no_default (sample 3 has credit=680), credit_score > 695 → default. This correctly labels sample 3 (no_default) at the cost of misclassifying high-credit defaulters with low weights.

✓ Step 4 complete. Stump 2 focuses on sample 3 using credit_score threshold, correctly classifying the previously-missed high-weight sample.

Step 5: Final Prediction

$f(x)={\alpha }_1{h}_1(x)+{\alpha }_2{h}_2(x)+\dots +{\alpha }_T{h}_T(x)$

$\hat{y}=\text{sign}(f(x))$

Each stump contributes proportionally to its weight ${\alpha }_t$. The final model is a weighted combination of all stumps — a "strong learner" built from many "weak learners."

✓ Step 5 complete. Final prediction is sign(Σαₜhₜ(x)). Weighted combination of all stumps, where each stump's influence scales with its accuracy.

Bagging vs Boosting — When Each Wins

Ensemble Vocabulary

When It Works and When It Doesn't

Bagging is your first choice when your base model has high variance — a deep decision tree that overfits, for instance. The less correlated the base models are, the more bagging helps. The limit: bagging cannot fix bias. If every tree underfits, averaging them still underfits.

Boosting is the tool for the opposite problem: when a shallow model systematically misses a region of the decision space. Each round forces the ensemble to focus on that missed region. The limit: boosting amplifies outliers. A single mislabeled sample gets upweighted every round until it dominates, and with more than about 5% label noise, boosting can degrade below a single model.

Neither method rescues a bad feature set. If your features carry no signal, no ensemble of weak models will find it.

Trace Table: Bagging and Boosting Phases

Related Concepts

Ensemble methods assume you already understand individual learners — specifically decision trees and how depth controls bias-variance tradeoff. If a single tree's behavior is unclear, the mechanics of combining them won't make sense. From here, bagging leads directly to Random Forest (which decorrelates bagged trees with feature subsampling), while boosting leads to AdaBoost (which formalizes the weight update as exponential loss minimization) and Gradient Boosting (which replaces sample reweighting with residual fitting).

Honest Limitations

Bagging reduces variance but leaves bias unchanged: bagging 100 high-bias stumps produces a high-bias ensemble. If your single model is already underfitting, adding more parallel copies doesn't help — switch to boosting or a more expressive base learner. Boosting is sensitive to noisy labels: a mislabeled sample gets upweighted every round until it dominates the weight distribution, corrupting all subsequent stumps. With more than ~5% label noise, boosting can degrade below a single tree. Neither method helps when the feature set is fundamentally wrong — no ensemble of weak learners recovers from completely uninformative features.

Test Your Understanding

1. The 63.2% unique sample rate for bootstrap is an asymptotic result: as $n\to \infty$, the probability that a specific sample is NOT drawn at least once is $(1-1/n{)}^n\to 1/e\approx 0.368$. For n=8 (our dataset), the exact probability that sample 0 is OOB is $(7/8{)}^8$. Compute this. Is it close to 36.8%?

2. In Step 3 of Boosting, we said sample 3 gets weight 0.501 after normalization. The 7 correct samples each get weight 0.071. Verify: $7\times 0.071+1\times 0.501=?$. Does it sum to 1? If there's a small discrepancy, where does it come from?

3. Bagging reduces variance but not bias. If the base learner is a depth-1 stump (already high bias), does bagging 100 stumps reduce the bias? Why or why not — and which method would you use instead?

4. Boosting can overfit: if you add a 100th stump that perfectly classifies all remaining noise, later stumps upweight noise samples and start memorizing them. What hyperparameter controls this in practice? What happens to training accuracy vs test accuracy as this hyperparameter increases?

5. The ensemble formula is $f(x)={\sum }_t{\alpha }_t{h}_t(x)$. Stump 1 has ${\alpha }_1=0.973$, stump 2 has ${\alpha }_2=0.641$. For a new sample where both stumps agree (both predict default=1), what is $f(x)$? What would $f(x)$ need to exceed for sign$(f(x))$ to predict default?