Decision Tree Regression

A classification tree uses entropy or Gini to measure node impurity. A regression tree uses variance — the same algorithmic skeleton, a different objective. At each leaf, instead of a majority vote, it predicts the mean of the samples that reached it.

Anchor dataset: 6 houses with sq_ft and price — the same anchor from the linear regression section.

import numpy as np

X = np.array([650, 850, 1100, 1400, 1600, 1900]).reshape(-1, 1)
y = np.array([180, 220, 280, 340, 370, 430])
# True OLS: ŷ = 53.33 + 0.20×sq_ft (from linear regression section)

Regression Trees vs Classification Trees

Everything else — threshold search, stopping conditions, pruning — is identical.

Root Node Variance

The variance at a node measures how spread out the target values are:

$\text{Var}(S)=\frac{1}{n}{\sum }_i(y_i-\bar{y}{)}^2$

At the root: $\bar{y}=(180+220+280+340+370+430)/6=1820/6=303.33$

$\text{Var}(\text{root})=\frac{(180-303.33{)}^2+(220-303.33{)}^2+(280-303.33{)}^2+(340-303.33{)}^2+(370-303.33{)}^2+(430-303.33{)}^2}{6}$

$=\frac{15210.9+6943.9+544.3+1344.3+4444.9+16045.3}{6}=\frac{44533.6}{6}=7422.3$

Threshold Search for sq_ft

Split criterion for regression:

${\text{IG}}_{\text{reg}}(S,t)=\text{Var}(S)-\frac{|S_l|}{|S|}\text{Var}(S_l)-\frac{|S_r|}{|S|}\text{Var}(S_r)$

5 midpoint candidates: 750, 975, 1250, 1500, 1750.

Best split: $t=1250$ with ${\text{IG}}_{\text{reg}}=6078$. The split at sq_ft = 1250 reduces the total variance from 7422 to 1344 — an 82% reduction.

Level 1: Left Node (sq_ft ≤ 1250, samples [650, 850, 1100])

$y=[180,220,280]$, $\bar{y}=226.7$, $\text{Var}=1555$.

Test thresholds $t=750$ and $t=975$:

• $t=750$: Left=[180], Var=0; Right=[220,280], $\bar{y}=250$, Var=900. Weighted: $(1/3)(0)+(2/3)(900)=600$. ${\text{IG}}_{\text{reg}}=1555-600=955$.
• $t=975$: Left=[180,220], $\bar{y}=200$, Var=400; Right=[280], Var=0. Weighted: $(2/3)(400)+(1/3)(0)=267$. ${\text{IG}}_{\text{reg}}=1555-267=1288$.

Best: $t=975$. Splits into:

• Left-Left (sq_ft ≤ 975): [650, 850] → $\bar{y}=200$ — predict $200k
• Left-Right (975 < sq_ft ≤ 1250): [1100] → $\bar{y}=280$ — predict $280k

Level 1: Right Node (sq_ft > 1250, samples [1400, 1600, 1900])

$y=[340,370,430]$, $\bar{y}=380$, $\text{Var}=1133$.

Test thresholds $t=1500$ and $t=1750$:

• $t=1500$: Left=[340], Var=0; Right=[370,430], $\bar{y}=400$, Var=900. Weighted: $(1/3)(0)+(2/3)(900)=600$. ${\text{IG}}_{\text{reg}}=1133-600=533$.
• $t=1750$: Left=[340,370], $\bar{y}=355$, Var=225; Right=[430], Var=0. Weighted: $(2/3)(225)+(1/3)(0)=150$. ${\text{IG}}_{\text{reg}}=1133-150=983$.

Best: $t=1750$. Splits into:

• Right-Left (1250 < sq_ft ≤ 1750): [1400, 1600] → $\bar{y}=355$ — predict $355k
• Right-Right (sq_ft > 1750): [1900] → $\bar{y}=430$ — predict $430k

The 4-Leaf Staircase

The orange staircase shows tree predictions: constant within each leaf region. The blue dashed line is the linear regression fit. For this near-linear dataset, the linear model tracks the data better; the tree's staircase has visible errors at the leaf boundaries.

Predictions vs Actual — Tree vs Linear

$\text{Tree MSE}=(400+400+0+225+225+0)/6=208.3$ $\text{Linear MSE}=(9+9+49+49+9+9)/6=22.3$

Linear regression wins decisively on this near-linear dataset. The tree is limited by its piecewise-constant prediction: leaves 1 and 4 each contain 2 samples whose true values differ, forcing the average to miss both.

sklearn Implementation

from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_squared_error

dt_reg = DecisionTreeRegressor(criterion='squared_error', max_depth=2, random_state=42)
dt_reg.fit(X, y)

y_pred = dt_reg.predict(X)
print(f"Tree MSE: {mean_squared_error(y, y_pred):.2f}")
print(f"Tree R²:  {dt_reg.score(X, y):.4f}")
print(f"Predictions:     {y_pred}")
print(f"Unique leaf predictions: {np.unique(y_pred)}")

The 4 unique prediction values are the mean of each leaf: [200, 280, 355, 430]. Despite high R²=0.972, the MSE of 208 is 9× worse than the linear model's MSE of 22.

max_depth Effect on Regression

print(f"{'depth':>8} {'MSE':>10} {'steps':>8} {'leaves':>8}")
for d in [1, 2, 3, None]:
    dt = DecisionTreeRegressor(max_depth=d, random_state=42)
    dt.fit(X, y)
    y_p = dt.predict(X)
    mse = mean_squared_error(y, y_p)
    steps = len(np.unique(y_p))
    print(f"{str(d):>8} {mse:>10.2f} {steps:>8} {dt.get_n_leaves():>8}")

At depth=3 with 6 samples and 6 leaves: each sample has its own leaf, MSE=22.22 (interpolation errors from single-sample leaves). At depth=None: MSE=0 (perfect memorization). Neither of these generalizes.

When Does a Regression Tree Beat Linear Regression?

The staircase is piecewise constant — it assumes the target is flat within each region. Linear regression assumes a global linear trend. Trees win when:

• The true relationship has sharp breakpoints (e.g., a salary cap at a specific experience level)
• The relationship is nonlinear with different slopes in different regions
• There are strong feature interactions (the effect of feature A depends on feature B)

Linear regression wins when the true relationship is approximately linear (as here).

Test Your Understanding

1. At the root, the best threshold was $t=1250$ with ${\text{IG}}_{\text{reg}}=6078$. The threshold $t=750$ produced ${\text{IG}}_{\text{reg}}=2902$. The left node of $t=750$ is pure (1 sample, Var=0), yet its IG is lower. Why does a single-sample pure left node not maximize variance reduction?

2. Leaf 1 (sq_ft ≤ 975) predicts $200kforbothhousesat650s{q}_{f}t(true$180k) and 850 sq_ft (true $220k).Thepredictionerroris$\pm$20k for both. What would the prediction be if you used the median instead of the mean? Would this reduce or increase MSE on this leaf?

3. At depth=3, MSE=22.22 with 6 leaves for 6 samples. Is this the irreducible error of the model, or could a depth=4 tree (if allowed) reduce it further? What would the depth=4 tree's MSE be?

4. The tree with max_depth=None achieves MSE=0 on training data. If you added a 7th house with sq_ft=1100 and price=$290k(differentfromtrainingsample{3}^{\prime }s$280k), what would the depth=None tree predict for sq_ft=1100? How does the presence of two training samples at the same sq_ft affect the tree?

5. DecisionTreeRegressor uses MSE by default (criterion='squared_error'). An alternative is MAE (criterion='absolute_error'). The MAE-optimal leaf prediction is the median instead of the mean. For leaf 1 with $y=[180,220]$: the mean is 200 and the median is also 200 (average of two). For a leaf with $y=[180,220,1000]$ (one outlier): what is the mean vs median, and which leaf prediction minimizes MAE?