Feature Engineering
Intermediate 14 min read Feature Engineering

Categorical Encoding

Turn categories into numbers without lying

Categorical features become numbers via one-hot, ordinal, frequency, target and embedding encodings; the right choice balances dimensionality, ordinality and leakage risk.

categorical encodingone-hotordinaltarget encodingfrequency encodingdummy variables
Prerequisites: Missing Value Imputation
Before we start

What you will learn

  • Choose between nominal and ordinal encodings from the data semantics
  • Fit one-hot, ordinal, frequency and target encoders and know when each is appropriate
  • Implement target encoding safely inside cross-validation to avoid leakage
Introduction

What is this?

Categorical encoding maps non-numeric values to numbers. One-hot/dummy creates indicator columns (nominal); OrdinalEncoder assigns ranks (ordered); Frequency/Count encoding replaces labels with their counts; Target/Mean encoding replaces labels with the mean target per category (often smoothed). Choice trades off dimensionality, information, cardinality and leakage risk.

Motivation

Why do you need it?

  • Models compute on numbers, not strings — encoding is mandatory, not optional.
  • One-hot on high-cardinality columns explodes features; smart encodings keep matrices lean.
  • Ordinality is real (size S<M<L); encoding order mistakes bake fake distances into the model.
  • Target encoding packs signal per column but leaks unless done fold-wise.
Think about it

Mental model

Imagine

Translating words between languages

Think of

Encoding categories to numbers

A naive translator picks the first dictionary match and loses tone; so can a naive encoder. Ordinality, rarity and meaning all matter — a 1:1 glossary is not a translation.

Mechanics

How it works

  • Inspect cardinality and semantics: nominal vs ordinal vs hierarchical.
  • Ordinal: use OrdinalEncoder only for true orders; map explicitly, don't guess alphabetical order.
  • Nominal low-cardinality: one-hot via pd.get_dummies or OneHotEncoder(handle_unknown='ignore').
  • Nominal high-cardinality: frequency or target encoding, or embeddings for text-like codes.
  • Fit on train folds only; reuse the same mapping in production.

Encoding decision flow.

Semantics
Cardinality
Low
High
Leak check
Hands-on

Examples

One-hot vs frequency vs safe target encoding

One-hot vs frequency vs safe target encoding
import pandas as pd
import numpy as np
from sklearn.model_selection import KFold
 
df = pd.DataFrame({
    "city": ["nyc", "sf", "nyc", "la", "nyc", "sf", "la", "nyc"],
    "amount": [100, 40, 80, 20, 60, 55, 25, 90],
})
 
onehot = pd.get_dummies(df["city"], prefix="city")
 
freq = df["city"].value_counts(normalize=True)
df["city_freq"] = df["city"].map(freq)
 
smooth = 5.0
glob = df["amount"].mean()
df["city_target"] = 0.0
for train_idx, val_idx in KFold(5, shuffle=True, random_state=1).split(df):
    tr = df.iloc[train_idx]
    va = df.iloc[val_idx]
    agg = tr.groupby("city")["amount"].agg(["mean", "count"])
    m = (agg["count"] * agg["mean"] + smooth * glob) / (agg["count"] + smooth)
    df.loc[va.index, "city_target"] = va["city"].map(m)
 
df["city_target_glob"] = df["city"].map(lambda c: (df.loc[df["city"]==c, "amount"].mean()))
 
print(df)
print(onehot.head())

One-hot stays safe for low cardinality; frequency encoding shrinks the matrix; and target encoding uses only out-of-fold means — the commented last line shows the leaking shortcut to never copy.

Real world

In production

  • Retail: store IDs (thousands of categories) → frequency + demand-target encodings beat 10k one-hot columns.
  • Insurance: code-level policy types → ordinal embedding captures hierarchy (auto -> collision -> liability).
  • NLP-lite: merchant names → target encoding with smoothing; one-hot would drown the model.
Guidelines

Best practices

  • Reserve adjacency for true order; always pass handle_unknown so unseen production categories don't crash; verify encoding columns count in train == prod; smooth target encoding with a global prior; nest every target-derived computation inside the CV loop.
Watch out

Common mistakes

  • Ordinal-encoding nominal categories with guessed order; one-hotting everything and blowing up memory; target-encoding on the full dataset (massive leakage); forgetting unseen categories in prod; relying on alphabetical orders from get_dummies.
When things break

Debugging tips

  • Diff the set of categories in train vs prod before scoring.
  • After one-hot, check pd.get_dummies column names match between train and live rows.
  • For target encoding, verify validation receives no train-category mean from its own rows.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

One-hot encoding a 500-category column produces how many columns?

2

Why should target encoding be fit inside CV folds?

Go deeper

Official documentation

sklearn.preprocessing.OneHotEncoder
docs
Target Encoding Done Right, 2026 (blog analysis)
video
pandas.get_dummies
docs