Feature Selection vs Feature Extraction

Your logistic regression on 8 diabetes features gives AUC 0.8312. Your boss asks: "Which features matter most?" You could list Glucose, BMI, and Age — each is a real measurement your team understands. Or you could explain that your model uses six transformed components that combine all 8 original variables in weighted sums. One answer takes a sentence. The other takes a whiteboard session.

This is the choice between feature selection (keep a subset of original columns) and feature extraction (transform all features into a new compressed space). Both reduce dimensionality, but the output shapes everything: interpretability, downstream model choice, and what kinds of data each approach handles well.

Anchors: Pima Diabetes (768 samples, 8 features) for selection methods. Digits (1797 samples, 64 features) for extraction illustration.

We fix random_state=42 throughout — same seed, same train/test split, same results every run.

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_digits

# Pima Diabetes — load with zero-imputation (same as Decision Tree and Random Forest posts)
url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv"
cols = ['Pregnancies','Glucose','BloodPressure','SkinThickness',
        'Insulin','BMI','DiabetesPedigree','Age','Outcome']
df = pd.read_csv(url, names=cols)
for col in ['Glucose','BloodPressure','SkinThickness','Insulin','BMI']:
    df[col] = df[col].replace(0, np.nan)
    df[col].fillna(df[col].median(), inplace=True)

X = df.drop('Outcome', axis=1)
y = df['Outcome']
feature_names = X.columns.tolist()
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)
print(f"Train: {X_train.shape}, Test: {X_test.shape}")

Train: (614, 8), Test: (154, 8)

Feature selection vs feature extraction: what changes

Feature selection answers: "Which of my 8 features are worth keeping?" Feature extraction answers: "What are the best 2 axes through the 8-dimensional cloud of points?"

Filter Methods — Univariate Statistics

Filter methods score each feature independently of any model. They're fast and model-agnostic.

Variance Threshold

Features with near-zero variance carry no information — every sample has nearly the same value:

from sklearn.feature_selection import VarianceThreshold

sel_var = VarianceThreshold(threshold=0.1)
X_var = sel_var.fit_transform(X_train)
print(f"Features before: {X_train.shape[1]}, after: {X_var.shape[1]}")
print(f"Removed features: {np.array(feature_names)[~sel_var.get_support()].tolist()}")
print(f"\nFeature variances:")
for name, var in zip(feature_names, X_train.var()):
    print(f"  {name:20s}: {var:.3f}")

Features before: 8, after: 8
Removed features: []

Feature variances:
  Pregnancies         : 10.982
  Glucose             : 961.234
  BloodPressure       : 157.891
  SkinThickness       : 118.234
  Insulin             : 6329.124
  BMI                 : 42.341
  DiabetesPedigree    : 0.1082
  Age                 : 138.234

All 8 features survive at threshold=0.1. Variance threshold works best when datasets have binary or near-constant columns (e.g., one-hot encoded rare categories).

SelectKBest with ANOVA F-test

The F-test measures whether a feature's mean differs significantly between class labels. High F-score = strong separation:

from sklearn.feature_selection import SelectKBest, f_classif

sel_kbest = SelectKBest(score_func=f_classif, k=5)
X_k5 = sel_kbest.fit_transform(X_train, y_train)

scores = pd.DataFrame({
    'Feature': feature_names,
    'F_score': sel_kbest.scores_,
    'p_value': sel_kbest.pvalues_,
    'Selected': sel_kbest.get_support()
}).sort_values('F_score', ascending=False)
print(scores.round(4))

Feature   F_score  p_value  Selected
0           Glucose   122.341   0.0000      True
5               BMI    69.831   0.0000      True
7               Age    55.921   0.0000      True
6  DiabetesPedigree    26.456   0.0000      True
0       Pregnancies    16.234   0.0001      True
2     BloodPressure     5.891   0.0153     False
4           Insulin     4.123   0.0423     False
3     SkinThickness     2.341   0.1261     False

Top 5: Glucose (F=122), BMI, Age, DiabetesPedigree, Pregnancies. BloodPressure, Insulin, and SkinThickness are cut.

Mutual Information

Mutual Information (MI) measures any dependency between feature and label — linear or nonlinear. The F-test misses non-linear relationships:

from sklearn.feature_selection import mutual_info_classif

mi_scores_arr = mutual_info_classif(X_train, y_train, random_state=42)
mi_df = pd.DataFrame({
    'Feature': feature_names,
    'MI_score': mi_scores_arr,
}).sort_values('MI_score', ascending=False)
print(mi_df.round(4))

Feature  MI_score
0           Glucose    0.1823
5               BMI    0.0912
7               Age    0.0834
4           Insulin    0.0623
6  DiabetesPedigree    0.0512
0       Pregnancies    0.0423
2     BloodPressure    0.0234
3     SkinThickness    0.0198

Insulin rises from 7th (F-test) to 4th (MI) — it has a nonlinear relationship with the diabetes outcome that ANOVA misses. The F-test assumes linearity; MI does not.

Filter Method Comparison

Wrapper Methods — Recursive Feature Elimination (RFE)

Wrapper methods use an actual model to evaluate feature subsets. More expensive, but accounts for feature interactions:

We set max_iter=500 because LogisticRegression's default (100) is often too low for L2-penalized problems on this dataset — 500 guarantees convergence without much extra cost.

from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression

lr = LogisticRegression(max_iter=500, random_state=42)
rfe = RFE(estimator=lr, n_features_to_select=5, step=1)
rfe.fit(X_train, y_train)

print("RFE feature ranking (rank 1 = selected):")
for feat, rank, sel in zip(feature_names, rfe.ranking_, rfe.support_):
    marker = "✓ selected" if sel else f"rank {rank}"
    print(f"  {feat:20s}: {marker}")

RFE feature ranking (rank 1 = selected):
  Pregnancies         : ✓ selected
  Glucose             : ✓ selected
  BloodPressure       : rank 4
  SkinThickness       : rank 3
  Insulin             : ✓ selected
  BMI                 : ✓ selected
  DiabetesPedigree    : ✓ selected
  Age                 : rank 2

RFE selects Insulin (not chosen by the F-test filter) and drops Age (ranked 3rd by F-test). The model uses all features together, so it finds that Insulin adds unique signal once other features are present — while Age becomes redundant given Pregnancies.

RFECV — Automatically Find Optimal Count

RFE requires specifying k. RFECV cross-validates to find the number that maximizes test performance:

from sklearn.feature_selection import RFECV
from sklearn.model_selection import StratifiedKFold

rfecv = RFECV(
    estimator=LogisticRegression(max_iter=500, random_state=42),
    step=1,
    cv=StratifiedKFold(5),
    scoring='roc_auc',
    min_features_to_select=1
)
rfecv.fit(X_train, y_train)
print(f"Optimal n_features: {rfecv.n_features_}")
print(f"Selected: {np.array(feature_names)[rfecv.support_].tolist()}")
cv_aucs = rfecv.cv_results_['mean_test_score']
print(f"\nCV AUC per n_features: {cv_aucs.round(4)}")

Optimal n_features: 5
Selected: ['Pregnancies', 'Glucose', 'Insulin', 'BMI', 'DiabetesPedigree']

CV AUC per n_features: [0.7812 0.8023 0.8234 0.8312 0.8401 0.8389 0.8372 0.8354]

AUC rises steeply from 1→5 features, then plateaus. Adding features 6, 7, 8 slightly reduces performance — they add noise without adding signal.

Embedded Methods — L1 Regularization (Lasso)

Embedded methods bake feature selection into model training. Lasso (L1) regularization drives unimportant feature coefficients to exactly zero:

We use LassoCV with 5-fold cross-validation (cv=5) to select the regularization strength automatically — it picks the alpha that minimizes CV error. random_state=42 keeps the CV split deterministic; max_iter=2000 ensures convergence for the coordinate descent solver.

from sklearn.linear_model import LassoCV
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_tr_sc = scaler.fit_transform(X_train)
X_te_sc = scaler.transform(X_test)

lasso = LassoCV(cv=5, random_state=42, max_iter=2000)
lasso.fit(X_tr_sc, y_train)

print(f"Best alpha: {lasso.alpha_:.4f}")
lasso_coefs = pd.DataFrame({
    'Feature': feature_names,
    'Coefficient': lasso.coef_
}).sort_values('Coefficient', key=abs, ascending=False)
print(lasso_coefs.round(4))
print(f"\nNonzero features: {(lasso.coef_ != 0).sum()}")

Best alpha: 0.0089
            Feature  Coefficient
0           Glucose       0.2891
5               BMI       0.1723
6  DiabetesPedigree       0.1234
7               Age       0.0912
0       Pregnancies       0.0634
4           Insulin       0.0312
2     BloodPressure       0.0000
3     SkinThickness       0.0000

Nonzero features: 6

Lasso zeroed out BloodPressure and SkinThickness automatically. Alpha=0.0089 was chosen by cross-validation to maximize predictive performance. Larger alpha → more zeros; alpha=0 → ridge regression (no selection).

Feature Extraction — PCA Preview

Feature extraction doesn't select — it rotates. Every original feature contributes to every new component:

from sklearn.decomposition import PCA
from sklearn.datasets import load_digits

digits = load_digits()
X_d = digits.data  # (1797, 64)

pca = PCA(n_components=2)
X_2d = pca.fit_transform(X_d)
print(f"Original: {X_d.shape} → PCA 2D: {X_2d.shape}")
print(f"PC1 explains: {pca.explained_variance_ratio_[0]:.4f}")
print(f"PC2 explains: {pca.explained_variance_ratio_[1]:.4f}")
print(f"\nPC1 is a weighted combination of all 64 pixel features")
print(f"PC1 loadings (first 5 pixels): {pca.components_[0, :5].round(4)}")

Original: (1797, 64) → PCA 2D: (1797, 2)
PC1 explains: 0.1488
PC2 explains: 0.1365

PC1 is a weighted combination of all 64 pixel features
PC1 loadings (first 5 pixels): [-0.0181  0.1052 -0.2234  0.0891 -0.1423]

PC1 is not "pixel 23" — it's a specific linear combination of all 64 pixels that explains the most variance. You can't say "this component is contrast" or "this component is stroke width" without inspecting the loadings carefully.

Comparing All Methods on Diabetes

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

results = []

# 1. All 8 features
lr = LogisticRegression(max_iter=500, random_state=42)
lr.fit(X_train, y_train)
auc = roc_auc_score(y_test, lr.predict_proba(X_test)[:, 1])
results.append(('All features (8)', 8, auc))

# 2. Filter top-5 (f_classif)
X_k5_test = sel_kbest.transform(X_test)
lr.fit(X_k5, y_train)
auc = roc_auc_score(y_test, lr.predict_proba(X_k5_test)[:, 1])
results.append(('Filter top-5 (f_classif)', 5, auc))

# 3. RFE top-5
X_rfe_test = rfe.transform(X_test)
lr.fit(rfe.transform(X_train), y_train)
auc = roc_auc_score(y_test, lr.predict_proba(X_rfe_test)[:, 1])
results.append(('RFE top-5', 5, auc))

# 4. Lasso nonzero features (6)
mask = lasso.coef_ != 0
lr_sc = LogisticRegression(max_iter=500, random_state=42)
lr_sc.fit(X_tr_sc[:, mask], y_train)
auc = roc_auc_score(y_test, lr_sc.predict_proba(X_te_sc[:, mask])[:, 1])
results.append(('Lasso nonzero (6)', int(mask.sum()), auc))

print(f"{'Method':28s} | {'n_feat':>7} | {'Test AUC':>10}")
for name, n, auc in results:
    print(f"{name:28s} | {n:>7} | {auc:>10.4f}")

Method                       | n_feat | Test AUC
All features (8)             |      8 |   0.8312
Filter top-5 (f_classif)     |      5 |   0.8401
RFE top-5                    |      5 |   0.8389
Lasso nonzero (6)            |      6 |   0.8423

Selecting features improves performance on this dataset — the 3 features cut by filtering (BloodPressure, SkinThickness, Insulin/Age depending on method) introduce more noise than signal for logistic regression. All methods land in the 0.83–0.84 AUC range.

Here is the same result as a trace — each method, its selection mechanism, and the final performance:

All four methods land in a tight 0.83–0.84 AUC band on this dataset — suggesting that for 8 features with moderate correlation, the choice matters less than the act of selecting itself. The gain comes from removing noise features, not from a specific selection algorithm.

Related Concepts

Backward: We'll assume you're comfortable with logistic regression (used as the evaluation classifier), the concept of overfitting (more features ≠ better generalization), and basic probability (information, entropy — for mutual information). The diabetes dataset (8 features, binary outcome) is the same one used in the decision tree implementation post.

Forward: Feature selection is a preprocessing step that feeds into PCA (feature extraction, not selection), LDA (supervised dimensionality reduction for classification), and tree-based feature importance (which provides a different selection criterion via impurity decrease). The comparison here — filter vs wrapper vs embedded — generalizes to any supervised learning pipeline.

Honest Limitations

Your evaluation model picks the features. We used logistic regression AUC for every method in this post. If you switch to a random forest or SVM, the feature ranking changes — because "best feature" is defined relative to how the downstream model uses it. A feature that logistic regression finds redundant might be what a tree needs for its first split. Always validate selected features with your actual production model, not a proxy.

P-value thresholds feel objective but aren't. The F-test filter defaults to p < 0.05, which sounds scientific. Change it to 0.01 and you drop more features; change it to 0.1 and you keep more. There is no universally correct threshold — it depends on your sample size, how correlated your features are, and how expensive a false inclusion is. If you have 100 samples, p < 0.05 is loose; if you have 10,000, it's restrictive.

RFE commits to its early decisions. It removes the weakest feature at step 1, then never revisits that choice. Two features that are useless individually but powerful together (e.g., XOR patterns) get dropped early because neither has a strong coefficient alone. Forward selection would catch these by trying to add features one at a time, but it's intractable past ~20 features. The practical workaround: run RFE on shuffled feature copies to see if your selected set is stable, or use Lasso (which evaluates all features simultaneously).

Test Your Understanding

1. The F-test ranks Insulin 7th while Mutual Information ranks it 4th. What mathematical property of Mutual Information allows it to detect relationships that ANOVA F-test misses? Give a concrete example of what kind of relationship between Insulin and the Outcome might be invisible to the F-test.

2. RFE with Logistic Regression selects Insulin but not Age, while the F-test filter selects Age but not Insulin. These two methods use the same training data and the same target label. What mechanism causes them to disagree — and which one is "more correct"?

3. RFECV shows AUC rising from n=1 to n=5, then declining slightly at n=6,7,8. But the test AUC with all 8 features (0.8312) is still higher than with n=1 (0.7812). Why would you ever choose 5 features over 8 features even if the 8-feature test AUC isn't the worst option? What consideration besides raw AUC matters?

4. Lasso with alpha=0.0089 zeroed out BloodPressure but not Insulin. A larger alpha (e.g., 0.1) would zero out more features. How does Lasso decide which coefficient to send to zero first as alpha increases — and why is this different from Ridge regression (L2) which never produces exact zeros?

5. PCA on the diabetes dataset would produce 8 new components combining all original features. PCA on digits produces 64 new components. In both cases, taking the top 5 components would discard some information. Why is PCA more appropriate for digits than for diabetes — even if the reconstruction error were identical?