Random Forest: Feature Importance and Feature Engineering

A trained Random Forest carries embedded feature importance scores — a side effect of building trees. This post covers two ways to extract them (impurity-based MDI and permutation importance), explains when each fails, and shows how to use them for feature selection and feature engineering.

Anchor dataset: Pima Indians Diabetes — 768 samples, 8 features (same preprocessing as Decision Tree post 06).

import pandas as pd
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

columns = ['Pregnancies','Glucose','BloodPressure','SkinThickness',
           'Insulin','BMI','DiabetesPedigree','Age','Outcome']
df = pd.read_csv('pima-indians-diabetes.csv', names=columns)

# Zero imputation (same as Decision Tree post)
zero_cols = ['Glucose','BloodPressure','SkinThickness','Insulin','BMI']
df[zero_cols] = df[zero_cols].replace(0, np.nan)
for col in zero_cols:
    df[col].fillna(df[col].median(), inplace=True)

X = df.drop('Outcome', axis=1)
y = df['Outcome']
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

Impurity-Based Feature Importance (MDI)

Mean Decrease in Impurity (MDI) accumulates the total Gini reduction each feature causes, weighted by the number of samples reaching each split, averaged across all trees:

$\text{importance}(j)=\frac{1}{T}{\sum }_{t=1}^T{\sum }_{s\in S_t(j)}\frac{n_s}{n}\cdot \Delta \text{Gini}(s)$

Where $S_t(j)$ = set of splits using feature $j$ in tree $t$, $n_s$ = samples at node $s$, $n$ = total training samples.

Manual sketch for Tree 1:

• Root split on Glucose ≤ 127.5: $n_s=614$ (all training samples), $\Delta \text{Gini}=0.15$.
  Weighted contribution: $614/614\times 0.15=0.150$.
• BMI split at level 1 (left branch): $n_s=380$, $\Delta \text{Gini}=0.08$.
  Weighted contribution: $380/614\times 0.08=0.0495$.

Summing across all splits in all 100 trees and dividing by T=100 gives MDI.

rf = RandomForestClassifier(n_estimators=100, max_features='sqrt', random_state=42, n_jobs=-1)
rf.fit(X_train, y_train)

importances = pd.Series(rf.feature_importances_, index=X.columns).sort_values(ascending=False)
print("Feature Importances (MDI):")
print(importances.round(4))

Glucose (28%) dominates — consistent with the Decision Tree single-model result. MDI from 100 trees is more stable than a single tree's importance because it averages across many different bootstrap samples.

MDI Weakness: Bias Toward High-Cardinality Features

MDI overestimates the importance of continuous features (many unique values) and underestimates binary/low-cardinality features. Why: continuous features offer more possible split thresholds → more opportunities to be selected → accumulate more total impurity reduction even if they aren't truly informative.

For example, a random float column (pure noise) would rank higher in MDI than a truly predictive binary column just because it offers 768 unique split points instead of 2.

Permutation Importance: Model-Agnostic Fix

Permutation importance measures the actual drop in test accuracy when a feature's values are randomly shuffled — breaking its relationship with the target. If accuracy drops a lot: the feature is important. If accuracy barely changes: the model didn't depend on it.

from sklearn.inspection import permutation_importance

perm = permutation_importance(rf, X_test, y_test,
                               n_repeats=30, random_state=42, n_jobs=-1)
perm_mean = pd.Series(perm.importances_mean, index=X.columns)
perm_std  = pd.Series(perm.importances_std, index=X.columns)

perm_series = perm_mean.sort_values(ascending=False)
print("Permutation Importance (mean ± std):")
for feat in perm_series.index:
    print(f"  {feat:20s}: {perm_mean[feat]:.4f} ± {perm_std[feat]:.4f}")

BloodPressure (0.008 ± 0.004) and SkinThickness (0.002 ± 0.003) both have standard deviations overlapping zero. Features where importance ± std crosses zero are not reliably contributing — their apparent importance may be noise. MDI ranked these differently, but permutation importance on held-out data gives the honest answer.

MDI vs Permutation Importance

Feature Selection: SelectFromModel

from sklearn.feature_selection import SelectFromModel

selector = SelectFromModel(rf, threshold='mean')
X_train_sel = selector.fit_transform(X_train, y_train)
X_test_sel  = selector.transform(X_test)

selected = X.columns[selector.get_support()].tolist()
print(f"Selected features ({len(selected)}): {selected}")

rf_all = RandomForestClassifier(n_estimators=100, random_state=42)
rf_sel = RandomForestClassifier(n_estimators=100, random_state=42)
rf_all.fit(X_train, y_train)
rf_sel.fit(X_train_sel, y_train)

print(f"All features ({X.shape[1]}):  {rf_all.score(X_test, y_test):.4f}")
print(f"Selected ({len(selected)}):           {rf_sel.score(X_test_sel, y_test):.4f}")

Removing the 3 weakest features (Insulin, BloodPressure, SkinThickness) slightly improves test accuracy (0.7727 → 0.7792). Noisy features add variance to the trees without adding signal — removing them cleans up the splits.

Feature Selection: RFECV

from sklearn.feature_selection import RFECV
from sklearn.model_selection import StratifiedKFold

rfecv = RFECV(
    estimator=RandomForestClassifier(n_estimators=50, random_state=42),
    step=1, cv=StratifiedKFold(5), scoring='roc_auc', n_jobs=-1
)
rfecv.fit(X_train, y_train)
print(f"Optimal n_features: {rfecv.n_features_}")
print(f"Selected: {X.columns[rfecv.support_].tolist()}")

RFECV independently confirms 5 features — same set as SelectFromModel, selected by CV rather than a threshold on importance scores.

Feature Engineering Guided by RF

After identifying Glucose and BMI as the two dominant predictors (both above 0.15 MDI), create interaction features:

X_eng = X.copy()
X_eng['Glucose_x_BMI'] = X['Glucose'] * X['BMI']
X_eng['Glucose_sq']    = X['Glucose'] ** 2

from sklearn.model_selection import cross_val_score

X_tr_all, X_te_all, y_tr, y_te = train_test_split(X_eng, y, test_size=0.2, random_state=42, stratify=y)

rf_base = RandomForestClassifier(n_estimators=100, random_state=42)
rf_eng  = RandomForestClassifier(n_estimators=100, random_state=42)

auc_base = cross_val_score(rf_base, X_train, y_train, cv=5, scoring='roc_auc').mean()
auc_eng  = cross_val_score(rf_eng,  X_tr_all, y_tr,   cv=5, scoring='roc_auc').mean()

print(f"Baseline AUC:          {auc_base:.4f}")
print(f"With interaction AUC:  {auc_eng:.4f}")

Check the importance of the new engineered features:

rf_eng.fit(X_tr_all, y_tr)
imp_eng = pd.Series(rf_eng.feature_importances_, index=X_eng.columns).sort_values(ascending=False)
print(imp_eng.round(4))

Glucose_x_BMI ranks 3rd (0.11) — above Age and DiabetesPedigree. The interaction captures cases where high glucose AND high BMI combine for elevated risk, beyond what each feature captures independently.

Test Your Understanding

1. MDI formula is: $\text{importance}(j)=\frac{1}{T}{\sum }_t{\sum }_{s\in S_t(j)}\frac{n_s}{n}\Delta \text{Gini}(s)$. A feature that appears only at depth-10 splits (where $n_s$ is small) vs a feature used at the root (where $n_s=n$). Which gets a higher MDI score per split? Why does depth systematically affect MDI?

2. Permutation importance shuffles a feature 30 times and averages the accuracy drop. If two features are highly correlated (say, Glucose and Glucose_sq with r=0.98), what happens to the permutation importance of each when one is shuffled — the other still carries the signal. Which of the two gets most of the permutation importance? How should you handle correlated features in feature selection?

3. SelectFromModel with threshold='mean' selects features with importance above the average. If one feature has MDI=0.90 (extremely dominant), all others have MDI≈0.01. What does this do to the mean threshold? Would most features be selected or dropped? What threshold might you use instead?

4. RFECV eliminated BloodPressure and SkinThickness. But a single deep decision tree post showed BloodPressure at 6% importance. How can RFECV and a single tree disagree about the same feature's value? What does CV in RFECV add that a single-tree importance doesn't?

5. The interaction feature Glucose_x_BMI ranked 3rd with MDI=0.11. But this feature is a function of Glucose and BMI — it can't carry information that isn't already available from the original features. Why does Random Forest benefit from explicit interaction features even though trees can theoretically capture interactions through sequential splits on the two features?