Feature Engineering
Intermediate 12 min read Feature Engineering

Scaling and Normalization

Put every feature on a comparable ruler

StandardScaler, MinMaxScaler and RobustScaler rescale numeric features for distance- and gradient-based models; trees are scale-agnostic, but scaling is not optional for the rest.

scalingnormalizationstandardizationz-scoreMinMaxRobustScalergradient descent
Prerequisites: Categorical Encoding
Before we start

What you will learn

  • Explain why scale-free algorithms (trees) don't need it while linear/neural/neighbor models do
  • Choose StandardScaler vs MinMaxScaler vs RobustScaler from data shape
  • Fit scalers on train and reuse the EXACT stats at inference
Introduction

What is this?

Scaling standardizes the magnitude of numeric features. StandardScaler center (mean 0, unit variance), MinMaxScaler maps to [0,1] using min/max, RobustScaler uses median + IQR to resist outliers. It changes geometry (distances, gradients) but not monotonic information, which is why tree-based models are indifferent to it.

Motivation

Why do you need it?

  • KNN and SVM's distance in raw units lets a wide feature (income in $) dwarf a narrow one (age in years).
  • Gradient descent converges faster and more stably when every feature step is comparable in rate.
  • Regularization (L1/L2) penalizes coefficient magnitude; unscaled scales distort what gets penalized.
  • Neural-net weight initialization assumes inputs near zero-mean unit-variance.
Think about it

Mental model

Imagine

Measuring rooms with a mix of meters, feet and yards

Think of

Modeling on unscaled features

You'd convert to one unit before comparing volumes. Scaling is that unit conversion for the model's arithmetic.

Mechanics

How it works

  • StandardScaler: (x - mean) / std — the default for linear models, PCA, SVMs, neural nets.
  • MinMaxScaler: (x - min) / (max - min) — preserves bounded ranges like pixel intensity, useful for images.
  • RobustScaler: (x - median) / IQR — for columns with angry outliers (income, sensor spikes).
  • Fit on training folds; persist the scaler and apply the identical transform to valid/test/prod data.
  • Trees (RandomForest, GBMs) and gradient-boosted trees ignore monotone rescalings — skip it there.

Scaling families.

Bounded
Gaussian
Outliers
Trees
Hands-on

Examples

Three scalers on a skewed feature

Three scalers on a skewed feature
import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler
 
data = np.array([[1.0], [2.0], [3.0], [100.0], [101.0]])  # heavy skew
df = pd.DataFrame(data, columns=["salary"])
 
std = StandardScaler().fit_transform(data).ravel()
mm = MinMaxScaler().fit_transform(data).ravel()
rob = RobustScaler().fit_transform(data).ravel()
 
res = pd.DataFrame({"raw": data.ravel(), "standard": std, "minmax": mm, "robust": rob})
print(res.round(3))
print("compare outlier impact on range:",
      (res.standard.max() - res.standard.min()).round(2),
      (res.robust.max() - res.robust.min()).round(2))

StandardScaler is dragged by the 100/101 outliers; RobustScaler anchors on the median, so the bulk of the data spans a comparable range and the outliers still shine through.

Real world

In production

  • Recommendation embeddings: cosine similarity on scaled vectors keeps magnitudes from dominating direction.
  • Credit scorecards: StandardScaler so L2-regularised logistic coefficients stay comparable across features.
  • Image models: MinMax to [0,1] feeds sigmoid-friendly activation pipelines.
Guidelines

Best practices

  • Scale only numeric model inputs, not the target (or log-transform the target instead); persist the fitted scaler object alongside the model (joblib); never refit scalers at inference on live batches; for tree ensembles, don't bother.
Watch out

Common mistakes

  • Scaling the target for a classifier's probability calibration; fitting MinMax on a column with outliers so the normal range collapses to 1px sliver; scaling after one-hot (dummy columns become non-0/1); scaling inside each fold inconsistently (fit per fold is fine, but always the same method).
When things break

Debugging tips

  • Print train and prod scale stats — big drift means a prod scaling bug.
  • Check for warnings about sparse input when scaling one-hot matrices.
  • Verify scaled data mean near 0 and std near 1 for StandardScaler on train.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which algorithm can safely skip scaling?

2

StandardScaler transform is?

Go deeper

Official documentation

sklearn.preprocessing - scaling
docs
Compare the effect of different scalers (sklearn example)
video
About feature scaling | Wikipedia
docs