Random Forest: Feature Importance and Feature Engineering

Your Random Forest model reports 79% accuracy on the diabetes test set. The stakeholder asks: "Which factors drive the prediction?" You know it's Glucose, BMI, maybe Age — but you need numbers to prove it. The model gives you feature importance scores for free (computed during training), but are they trustworthy? Glucose ranks highest in one measure, but Permutation Importance says BloodPressure and SkinThickness barely contribute. Which one do you believe?

That's the problem with feature importance: there are multiple definitions, each with different failure modes. Getting it wrong means dropping signal or keeping noise — both bad for production.

What Is Feature Importance?

Feature importance scores tell you how much each input feature contributes to your model's predictions. Random Forest gives you two flavors: MDI (Mean Decrease in Impurity) computed during training as a side effect of building trees, and Permutation Importance measured on held-out data by shuffling each feature and observing the accuracy drop. MDI is free but biased; Permutation Importance is honest but expensive. This post shows you when each works, when each fails, and how to use them together for feature selection and engineering.

Anchor dataset: Pima Indians Diabetes — 768 samples, 8 features (same preprocessing as Decision Tree post 06).

import pandas as pd
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

columns = ['Pregnancies','Glucose','BloodPressure','SkinThickness',
           'Insulin','BMI','DiabetesPedigree','Age','Outcome']
df = pd.read_csv('pima-indians-diabetes.csv', names=columns)

# Zero imputation (same as Decision Tree post)
zero_cols = ['Glucose','BloodPressure','SkinThickness','Insulin','BMI']
df[zero_cols] = df[zero_cols].replace(0, np.nan)
for col in zero_cols:
    df[col].fillna(df[col].median(), inplace=True)

X = df.drop('Outcome', axis=1)
y = df['Outcome']
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

Impurity-Based Feature Importance (MDI)

Mean Decrease in Impurity (MDI) accumulates the total Gini reduction each feature causes, weighted by the number of samples reaching each split, averaged across all trees:

$\text{importance}(j)=\frac{1}{T}{\sum }_{t=1}^T{\sum }_{s\in S_t(j)}\frac{n_s}{n}\cdot \Delta \text{Gini}(s)$

Where $S_t(j)$ = set of splits using feature $j$ in tree $t$, $n_s$ = samples at node $s$, $n$ = total training samples.

Manual sketch for Tree 1:

• Root split on Glucose ≤ 127.5: $n_s=614$ (all training samples), $\Delta \text{Gini}=0.15$.
  Weighted contribution: $614/614\times 0.15=0.150$.
• BMI split at level 1 (left branch): $n_s=380$, $\Delta \text{Gini}=0.08$.
  Weighted contribution: $380/614\times 0.08=0.0495$.

Summing across all splits in all 100 trees and dividing by T=100 gives MDI. random_state=42 ensures reproducible bootstrap samples and split decisions. n_jobs=-1 trains trees in parallel across all CPU cores.

rf = RandomForestClassifier(n_estimators=100, max_features='sqrt', random_state=42, n_jobs=-1)
rf.fit(X_train, y_train)

importances = pd.Series(rf.feature_importances_, index=X.columns).sort_values(ascending=False)
print("Feature Importances (MDI):")
print(importances.round(4))

Feature Importances (MDI):
Glucose            0.2831
BMI                0.1612
Age                0.1378
DiabetesPedigree   0.1089
Insulin            0.0831
Pregnancies        0.0923
BloodPressure      0.0712
SkinThickness      0.0624

Glucose (28%) dominates — consistent with the Decision Tree single-model result. MDI from 100 trees is more stable than a single tree's importance because it averages across many different bootstrap samples.

MDI Weakness: Bias Toward High-Cardinality Features

MDI overestimates the importance of continuous features (many unique values) and underestimates binary/low-cardinality features. Why: continuous features offer more possible split thresholds → more opportunities to be selected → accumulate more total impurity reduction even if they aren't truly informative.

For example, a random float column (pure noise) would rank higher in MDI than a truly predictive binary column just because it offers 768 unique split points instead of 2.

Permutation Importance: Model-Agnostic Fix

Permutation importance measures the actual drop in test accuracy when a feature's values are randomly shuffled — breaking its relationship with the target. If accuracy drops a lot: the feature is important. If accuracy barely changes: the model didn't depend on it.

from sklearn.inspection import permutation_importance

perm = permutation_importance(rf, X_test, y_test,
                               n_repeats=30, random_state=42, n_jobs=-1)
perm_mean = pd.Series(perm.importances_mean, index=X.columns)
perm_std  = pd.Series(perm.importances_std, index=X.columns)

perm_series = perm_mean.sort_values(ascending=False)
print("Permutation Importance (mean ± std):")
for feat in perm_series.index:
    print(f"  {feat:20s}: {perm_mean[feat]:.4f} ± {perm_std[feat]:.4f}")

Permutation Importance (mean ± std):
  Glucose             : 0.0915 ± 0.0121
  BMI                 : 0.0447 ± 0.0098
  Age                 : 0.0312 ± 0.0089
  DiabetesPedigree    : 0.0234 ± 0.0071
  Insulin             : 0.0189 ± 0.0065
  Pregnancies         : 0.0156 ± 0.0055
  BloodPressure       : 0.0078 ± 0.0044
  SkinThickness       : 0.0023 ± 0.0031

BloodPressure (0.008 ± 0.004) and SkinThickness (0.002 ± 0.003) both have standard deviations overlapping zero. Features where importance ± std crosses zero are not reliably contributing — their apparent importance may be noise. MDI ranked these differently, but permutation importance on held-out data gives the honest answer.

MDI vs Permutation Importance

Feature Selection: SelectFromModel

from sklearn.feature_selection import SelectFromModel

selector = SelectFromModel(rf, threshold='mean')
X_train_sel = selector.fit_transform(X_train, y_train)
X_test_sel  = selector.transform(X_test)

selected = X.columns[selector.get_support()].tolist()
print(f"Selected features ({len(selected)}): {selected}")

rf_all = RandomForestClassifier(n_estimators=100, random_state=42)
rf_sel = RandomForestClassifier(n_estimators=100, random_state=42)
rf_all.fit(X_train, y_train)
rf_sel.fit(X_train_sel, y_train)

print(f"All features ({X.shape[1]}):  {rf_all.score(X_test, y_test):.4f}")
print(f"Selected ({len(selected)}):           {rf_sel.score(X_test_sel, y_test):.4f}")

Selected features (5): ['Glucose', 'BMI', 'Age', 'DiabetesPedigree', 'Pregnancies']
All features (8):  0.7727
Selected (5):      0.7792

Removing the 3 weakest features (Insulin, BloodPressure, SkinThickness) slightly improves test accuracy (0.7727 → 0.7792). Noisy features add variance to the trees without adding signal — removing them cleans up the splits.

Feature Selection: RFECV

from sklearn.feature_selection import RFECV
from sklearn.model_selection import StratifiedKFold

rfecv = RFECV(
    estimator=RandomForestClassifier(n_estimators=50, random_state=42),
    step=1, cv=StratifiedKFold(5), scoring='roc_auc', n_jobs=-1
)
rfecv.fit(X_train, y_train)
print(f"Optimal n_features: {rfecv.n_features_}")
print(f"Selected: {X.columns[rfecv.support_].tolist()}")

Optimal n_features: 5
Selected: ['Pregnancies', 'Glucose', 'BMI', 'DiabetesPedigree', 'Age']

RFECV independently confirms 5 features — same set as SelectFromModel, selected by CV rather than a threshold on importance scores.

Feature Engineering Guided by RF

After identifying Glucose and BMI as the two dominant predictors (both above 0.15 MDI), create interaction features:

X_eng = X.copy()
X_eng['Glucose_x_BMI'] = X['Glucose'] * X['BMI']
X_eng['Glucose_sq']    = X['Glucose'] ** 2

from sklearn.model_selection import cross_val_score

X_tr_all, X_te_all, y_tr, y_te = train_test_split(X_eng, y, test_size=0.2, random_state=42, stratify=y)

rf_base = RandomForestClassifier(n_estimators=100, random_state=42)
rf_eng  = RandomForestClassifier(n_estimators=100, random_state=42)

auc_base = cross_val_score(rf_base, X_train, y_train, cv=5, scoring='roc_auc').mean()
auc_eng  = cross_val_score(rf_eng,  X_tr_all, y_tr,   cv=5, scoring='roc_auc').mean()

print(f"Baseline AUC:          {auc_base:.4f}")
print(f"With interaction AUC:  {auc_eng:.4f}")

Baseline AUC:          0.8304
With interaction AUC:  0.8341

Check the importance of the new engineered features:

rf_eng.fit(X_tr_all, y_tr)
imp_eng = pd.Series(rf_eng.feature_importances_, index=X_eng.columns).sort_values(ascending=False)
print(imp_eng.round(4))

Glucose            0.2311
BMI                0.1489
Glucose_x_BMI      0.1102
Age                0.1121
DiabetesPedigree   0.0945
Pregnancies        0.0812
Glucose_sq         0.0781
Insulin            0.0612
BloodPressure      0.0511
SkinThickness      0.0316

Glucose_x_BMI ranks 3rd (0.11) — above Age and DiabetesPedigree. The interaction captures cases where high glucose AND high BMI combine for elevated risk, beyond what each feature captures independently.

When It Works and When It Doesn't

Reach for MDI when you need a quick ranking during exploratory modeling — it's computed for free during training and gives you a rough sense of which features matter. Use Permutation Importance when the ranking needs to be trustworthy (publication, feature selection for production). Permutation Importance also wins when your data has mixed dtypes: it's unbiased toward continuous features.

The limit: both methods fail on highly correlated features. MDI splits importance between correlated pairs arbitrarily (the first split grabs most of the credit). Permutation Importance double-counts: shuffling one of two correlated features barely drops accuracy because the other carries the same signal. Neither method tells you about interaction effects directly — you still need domain knowledge to create interaction features, as shown in the feature engineering section above.

Trace Table: Feature Importance Methods on Pima Indians

Related Concepts

This post requires understanding how Random Forest builds trees (the previous post) and how Gini impurity measures split quality (from the Decision Tree series). MDI is computed as a byproduct of training — it reuses the impurity decrease that was already calculated at each split. Forward, the feature ranking techniques here feed directly into feature engineering decisions for any model, not just Random Forest: once you know which features carry signal, you can create interaction terms, drop noise, and reduce dimensionality before feeding data into gradient boosting or XGBoost.

Honest Limitations

MDI is unreliable on datasets with a mix of continuous and categorical features: a continuous feature with 500 unique values accumulates more split opportunities than a binary flag and will appear more important even if it adds no predictive value. With fewer than ~200 training samples, permutation importance standard deviations become very wide — a feature's importance interval will span zero even if it genuinely matters, making the ranking untrustworthy. RFECV is slow: with 5-fold CV and n_estimators=50, it trains 5 × 8 × 50 = 2000 trees to select 8 features — on large datasets, use SelectFromModel or a quick MDI ranking first to reduce candidate features before running RFECV.

Test Your Understanding

1. MDI formula is: $\text{importance}(j)=\frac{1}{T}{\sum }_t{\sum }_{s\in S_t(j)}\frac{n_s}{n}\Delta \text{Gini}(s)$. A feature that appears only at depth-10 splits (where $n_s$ is small) vs a feature used at the root (where $n_s=n$). Which gets a higher MDI score per split? Why does depth systematically affect MDI?

2. Permutation importance shuffles a feature 30 times and averages the accuracy drop. If two features are highly correlated (say, Glucose and Glucose_sq with r=0.98), what happens to the permutation importance of each when one is shuffled — the other still carries the signal. Which of the two gets most of the permutation importance? How should you handle correlated features in feature selection?

3. SelectFromModel with threshold='mean' selects features with importance above the average. If one feature has MDI=0.90 (extremely dominant), all others have MDI≈0.01. What does this do to the mean threshold? Would most features be selected or dropped? What threshold might you use instead?

4. RFECV eliminated BloodPressure and SkinThickness. But a single deep decision tree post showed BloodPressure at 6% importance. How can RFECV and a single tree disagree about the same feature's value? What does CV in RFECV add that a single-tree importance doesn't?

5. The interaction feature Glucose_x_BMI ranked 3rd with MDI=0.11. But this feature is a function of Glucose and BMI — it can't carry information that isn't already available from the original features. Why does Random Forest benefit from explicit interaction features even though trees can theoretically capture interactions through sequential splits on the two features?