Feature Engineering
Intermediate 14 min read Feature Engineering

Feature Selection

Cut noise, keep signal

Feature selection drops constant, correlated and low-information columns using filters, wrappers and embedded importance — shrinking models, cutting cost and limiting overfitting.

feature selectionVarianceThresholdSelectKBestRFEmutual informationcollinearity
Prerequisites: Feature Engineering Introduction
Before we start

What you will learn

  • Run filter (variance, correlation, mutual information), wrapper (RFE) and embedded (importance) selection
  • Detect and drop highly correlated feature groups
  • Size the selected set by validation score, not arbitrary k
Introduction

What is this?

Feature selection reduces the feature matrix by removing inputs that add no signal: near-constant columns, near-duplicates (collinear pairs), and low mutual-information features. Methods: filters (cheap, univariate), wrappers (RFE — fit/spit/refit), embedded (L1 penalty, tree importances). Fewer features mean faster, cheaper, more interpretable models with lower overfitting risk.

Motivation

Why do you need it?

  • Curse of dimensionality: with fixed samples, accuracy peaks at moderate cardinality then decays.
  • Duplicate/correlated features waste compute and inflate variance in linear coefficient estimates.
  • L1 and tree-importance selection automate the pruning with one fit.
  • A 50-feature model is vastly easier to audit and debug than a 5,000-feature one.
Think about it

Mental model

Imagine

Packing a backpack for a trail

Think of

Feature selection

Every extra item brings weight and clutter; a pack with the essentials alone is lighter, faster and less fragile. Leave the microwave behind.

Mechanics

How it works

  • Constant/half-constant: VarianceThreshold(threshold=0.0 or 0.05*max(variance)).
  • Collinearity: correlation matrix |rho|>0.9; drop one of the pair (prefer the more interpretable or cheaper).
  • Univariate filters: SelectKBest(mutual_info_classif/regression) — captures non-linear association too.
  • Wrappers: RFE fits the model repeatedly, pruning weakest-ranked features each round.
  • Embedded: L1 (Lasso) zeroes weights; tree Gini/mdi importance ranks columns.
  • Validate the cardinality by model score on holdout, not by importance rank alone.

Feature selection cascade.

Variance
Collinear
Mutual info
RFE / L1
Validate
Hands-on

Examples

Filter + RFE pipeline

Filter + RFE pipeline
import numpy as np
import pandas as pd
from sklearn.feature_selection import VarianceThreshold, SelectKBest, mutual_info_classif, RFE
from sklearn.linear_model import LogisticRegression
 
df = pd.read_csv("https://raw.githubusercontent.com/mwaskom/seaborn-data/master/titanic.csv")
 
df["sex"] = (df["sex"] == "male").astype(int)
df["age"] = df["age"].fillna(df["age"].median())
tab = pd.crosstab(df.index, df["embark_town"], prefix="town")
X = pd.concat([df[["fare", "age", "pclass", "sex"]], tab], axis=1).astype(float)
y = df["survived"].astype(int)
 
keep_var = VarianceThreshold(threshold=0.01).fit(X)
print("variance-kept cols:", X.columns[keep_var.get_support()].tolist())
 
kb = SelectKBest(mutual_info_classif, k=4).fit(X, y)
print("top-4 by mutual info:", X.columns[kb.get_support()].tolist())
 
rfe = RFE(LogisticRegression(max_iter=1000), n_features_to_select=3).fit(X, y)
print("RFE top-3:", X.columns[rfe.support_].tolist())
print("RFE ranking:", dict(zip(X.columns, rfe.ranking_)))

VarianceThreshold kills near-constant town columns, mutual information selects the strongest informative inputs, and RFE lets the actual model prune down to its best three.

Real world

In production

  • Churn: shrink 800 SaaS features to ~25 with mutual-info + L1 before the GLM team owns the model.
  • Fraud: RFE with boosted trees on 1,000+ velocity features lands at ~80 with identical AUC — 10x cheaper.
  • Medical: scorecard regulation demands few, interpretable features; wrappers find them.
Guidelines

Best practices

  • Select features in a CV-safe way (fit selectors on train folds only); combine a cheap filter first (variance/correlation) then the model-aware stage; re-run selection after data refreshes; log the selected set to the model card.
Watch out

Common mistakes

  • Selecting on the full dataset and leaking fold info; using variance on unscaled features so scale dominates; trusting tree importances blindly with collinear features (importances split arbitrarily between them); equating zero importance with 'drop it'.
When things break

Debugging tips

  • Compare importance spectra with and without correlated doubles.
  • After dropping columns, confirm the prod pipeline emits exactly the selected columns.
  • Overfit warning: if AUC drops sharply on validation vs train after selection, fold-leak or edge selection.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which selector best captures non-linear feature-target association?

2

Drop one of a |rho|>0.9 pair because:

Go deeper

Official documentation

sklearn.feature_selection
docs
Feature selection - scikit-learn example gallery
video
RFE - scikit-learn API
docs